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ABSTRACT 

This report is the first in a series on Project GRAD 
(Graduation Really Achieves Dreams) in the Newark, New Jersey Public Schools. 
Project GRAD is an education initiative that combines several proven or 
promising reforms with the goals of increasing reading and math achievement 
test scores, improving classroom behavior, reducing dropout rates, and 
increasing rates of college enrollment and graduation. It is designed to be 
implemented across a feeder pattern of schools and targets inner city schools 
with a history of low achievement and low graduation and college enrollment 
rates. This report presents evaluation data from the project’s implementation 
period (1998-99). Data from field research, teacher surveys and focus groups, 
and analysis of student data indicates that the partnership between Project 
GRAD and the Newark Public Schools made significant strides toward 
implementation in Newark in a short time period. Its college scholarship 
guarantee stimulated interest in and support for the initiative among school 
staff, students, parents, and community institutions. Implementation of 
Project GRAD met and exceeded planned expectations. Although neither the 
reading nor math components had yet been implemented, the program's other 
components were having a positive effect on some students' achievement. 
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Preface 



Project GRAD, conceived and first implemented in Houston, Texas, is a relatively new 
education initiative, but one that has already generated strong interest and been expanded to five 
other cities: Atlanta, Columbus, Lx)s Angeles, Nashville, and Newark. 

The interest in Project GRAD reflects the fact that the initiative’s core features directly 
address problems that plague urban school districts. Specifically, Project GRAD (1) recognizes 
the importance of effecting change at all school levels, and of affecting students throughout their 
school years, by focusing on a feeder pattern (that is, a high school and the middle and 
elementary schools that feed into it); (2) offers a comprehensive classroom management program 
to handle discipline issues and to build a shared sense of responsibility among the students and 
adults in a school building; (3) combines a nationally recognized reading program and a strong 
math component to enhance elementary school curricula; (4) brings volunteers, special activities, 
and social service programs into the schools to provide a safety net for students at all grade 
levels; (5) provides summer programming on college campuses so that high school youth are 
exposed to an enriched curriculum and gain early awareness of college choices; and (6) offers 
college scholarships to students who meet certain criteria with the aim of enhancing student 
engagement while promoting academic performance and college readiness. 

This first report from MDRC’s ongoing evaluation of Project GRAD Newark, the initial 
expansion site, is focused on early implementation of the initiative — its pre-curricular phase. 
With support from the Ford Foundation, the Lucent Technologies Foundation, and the Grable 
Foundation, and with the helpful assistance of the Newark Public Schools and the nonprofit 
entity Project GRAD Newark, Inc., MDRC began this evaluation in January 1998. The 
evaluation affords a view of Project GRAD’s components, principles, and implementation period 
as they are being adapted to a new context and offers policymakers important insight into the 
process of introducing a major education reform into a school system. 

The findings from the evaluation thus far suggest that the implementation of Project 
GRAD in Newark is off to a strong start, and there are promising signs of progress. Upcoming 
reports on Project GRAD Newark will focus on the initiative’s curricular components and their 
effects on student achievement outcomes. 



Judith M. Gueron 
President 
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Chapter 1 

Introduction 



This evaluation report is the first in a series on Project GRAD (Graduation Really 
Achieves Dreams) in the Newark, New Jersey Public Schools. Begun in Houston in 1993, Project 
GRAD is an education initiative that combines a number of proven or promising reforms with 
the goals of increasing reading and math achievement test scores, improving classroom behavior, 
reducing dropout rates, and increasing rates of college enrollment and graduation. Project GRAD 
is designed to be implemented across a feeder pattern of schools (that is, a high school and the 
middle and elementary schools that feed into it) and is targeted at inner-city schools with a his- 
tory of low academic achievement and low rates of graduation and college enrollment. This 
evaluation report focuses on the implementation of Project GRAD in Newark over the period 
from January 1998 to December 1999: the program’s launch (January 1998-August 1998), the 
first full program year (September 1998-June 1999), and the first semester of the second program 
year (September 1999-December 1999). The next report will cover the second full program year 
and the program’s impacts in more detail. 

The Project GRAD Newark initiative was launched in Newark in February 1998 and is 
being implemented in the Malcolm X Shabazz High School feeder pattern, which includes the 
high school, one middle school, and seven elementary schools serving a total of approximately 
6,500 mostly low-income students.' Five of the elementary schools also serve middle grades 6-8. 
(Chapter 2 provides more information on the Project GRAD Newark schools and students.) 

The Project GRAD model was designed by the former Chief Executive Officer of Ten- 
neco, James Ketelsen, to help inner-city schools effectively educate students and prepare them to 
succeed in college (for more background, see Box 1.1). By using a feeder pattern approach. Pro- 
ject GRAD ensures that multiple schools within each participating district are affected by its 
change process and that, in turn, a large number of students are provided with a consistent, 
achievement-oriented education over an extended period as they advance through school grades. 
Project GRAD draws on five program components and provides program coordination assis- 
tance, as described in greater detail in Box 1.2.^ Organizational partnerships are essential to the 
operation of Project GRAD: Typically, each Project GRAD initiative is managed by a private, 
nonprofit organization governed by a partnership comprising funders of the initiative, the school 
district, and other community organizations. This nonprofit entity provides administrative and 
resource stability for Project GRAD efforts, serving to buffer the initiative against the inevitable 
changes (for example, a change in district superintendent) that each partner organization will un- 
dergo. 



'This student enrollment figure is based on information obtained from Philadelphia Online’s School Report 
Cards (1998). 

^In some cases, a school district wishing to implement Project GRAD may opt to implement only four of the five 
Project GRAD program components. Substitution of one of the recommended components may be made in cases 
where the replacement program’s effectiveness is supported by research-based evidence comparable to that support- 
ing the substituted Project GRAD component. 
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Box 1.1 



Grounding Program Implementation in School Reform Research 

Improvement of America’s schools, particularly those located in poor, inner-city communities, has 
been a rallying cry for nearly two decades — a national priority given impetus by the report A Nation at 
Risk,^ Since then, approaches to school reform have targeted many areas: school-based decision- 
making, curricular standards, whole-school reform, and accountability; and the list continues to grow. 
Experiences from these efforts have led to important lessons and numerous models for implementing ef- 
fective school change.^ However, because the impacts of school reform models are shaped by individ- 
ual school environments, the process and outcomes of such efforts are inevitably different in each 
school. 

One obstacle to building consistency across schools is that many models are designed to address 
speciEc aspects of the educational system and fail to piece together a coherent strategy of teaching and 
learning within a single school, not to mention across the K-12 grade span. Competing demands on 
limited resources — such as discretionary funds, staff, technical assistance, and time — can hinder the 
development of strong education program linkages within a district. Clearly, the next step in the 
school-reform learning process is to develop models that are effective in improving multiple schools 
(that is, to scale up school reform) in low-performing school districts. 

Scaling up school change effectively is extremely difficult because it requires devoting attention 
to the entire K-12 grade span as well as the input and cooperation of many actors with different levels 
of authority. As a result, schools are often overwhelmed when trying to respond to complex directives 
from the state and district simultaneously. Scaling up school reform also means taking on teaching and 
learning in middle and high schools; high school, in particular, is often considered to be too far along in 
students’ development for school reform to produce change. Research has pointed to the need to get 
middle and high schools engaged seriously in reform to prepare youth adequately for college and the 
work force. The Department of Education recognizes this need and has furnished millions of dollars in 
support of comprehensive school reform models that include grades K-12. 

Project GRAD is one promising effort to address the operational tensions in current approaches to 
school reform. Launched in the fall of 1993 with the objective of creating an effective model for scaling 
up school change across the K-12 grade span. Project GRAD integrates a set of proven or promising re- 
forms. James Ketelsen, retired Chief Executive Officer of Tenneco, Inc., developed Project GRAD in 
collaboration with the Houston Independent School District (HISD) to encompass and extend school 
improvement efforts through a business-school partnership between Tenneco, Inc., and HISD’s Jeffer- 
son Davis High School that was established in 1981. The partnership provided Jefferson Davis students 
with academic Enancial incentives, enrichment, and support activities — including college scholarships, 
university-based summer institutes, mentoring and tutoring, social services, leadership development 
workshops, and summer jobs. 

Despite these advantages, however, Jefferson Davis students continued to struggle academically; 
extra academic resources at the high school level were offered too late to compensate for students’ ear- 
lier, and often less rigorous, academic preparation. Therefore, high school graduation and college en- 
rollment rates among Jefferson Davis students remained low. This experience highlighted the need to 
enhance academic learning at the middle and elementary school levels to enable students to take advan- 
tage of the college preparation and scholarship opportunities being offered to them, which prompted the 
creation of Project GRAD. 

^National Commission on Excellence in Education, 1983. 

^Bonnilly, 1998; Education Trust, 1999; Fuhrman, 1994; O’Day, Goertz, and Floden, 1995; Opuni, 1998; 
Slavin, 1997; and U.S. Department of Education, 1998. 
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Box 1.2 



Components of Project GRAD Newark 

Creating a School and Classroom Environment Conducive to Optimal Leamine 

The GRAD college scholarship guarantee is often referred to as the cornerstone of Project GRAD because it most 
directly represents the ultimate goal of the initiative: increasing student enrollment in college. Through this component. 
Project GRAD seeks to raise the academic expectations of students in grades Pre-K-12 by providing a financial incen- 
tive and college awareness, preparation, and recruitment activities. In Newark the GRAD scholarship is $6,000; $1,000 
is “earned” annually during the freshman and sophomore years of college and $2,000 is earned annually during the jun- 
ior and senior years. This scholarship is guaranteed to each graduate of a Project GRAD high school who meets the fol- 
lowing program requirements during his or her high school tenure: completes three years of math coursework (begin- 
ning with Algebra I), has at least a 2.5 four-year cumulative grade point average in core academic subjects at the time of 
graduation, completes two university-based summer academic institutes, and graduates within four years of freshman 
enrollment. 

Consistency Management & Cooperative Discipline (CMCD) is a research-based classroom management and dis- 
cipline program that seeks to build student support for classroom management, responsibility, and self-discipline by 
promoting cooperative learning and positive working relationships among students, teachers, and other adults in the 
school. Full-time, school-based CMCD staff developers are designated to assist school staff in incorporating CMCD 
practices into their daily school routine. 

Communities in Schools (CIS) is a program that brings additional support (that is, volunteers, social services, and 
academic enrichment and support activities) directly into schools. By placing a full-time CIS project director in each 
Project GRAD school, the CIS component seeks to enhance social and academic support services available through the 
school and to provide targeted assistance to students with problems outside school that affect their classroom perform- 
ance. 



Implementing Effective Curricular Reform in Reading and Math in the Elementary Grades 

Success for All (SFA) is a nationally recognized reading program that promotes comprehensive restructuring of 
most school resources to provide concentrated instructional time for reading to bring students to grade level in this sub- 
ject area by 3”^ grade. 

Math Opportunities, Valuable Experiences, Innovative Teaching (MOVE IT Math) is a K-6 professional devel- 
opment program that advocates math instruction based on the use of manipulatives to address a wide variety of learning 
styles (visual, auditory, kinesthetic). Project GRAD Houston uses MOVE IT Math, which is the math curriculum com- 
ponent of the standard Project GRAD model. The Project GRAD Newark, Inc., Board has approved the Newark school 
district’s recommendation to implement MathWings as the math curriculum component for Project GRAD Newark. 
MathWings provides students in grades 1-5 with instruction in basic math, problem-solving, and concept development. 
It is structured to accommodate all levels of mathematical ability. 

Enhancing School-Level Capacity for Program Management 

Apart from the GRAD scholarship component, all components of Project GRAD were stand-alone programs being 
implemented in other schools in Newark or elsewhere in the country prior to their integration under Project GRAD. 
Each component has a lead agency or developer that provides schools with materials, professional development, techni- 
cal assistance, and other resources specifically designed to support implementation of its program. However, Project 
GRAD provides program coordination assistance by securing high-quality staff to provide schools (especially princi- 
pals) with guidance in managing the initiative and its various components. 

The program coordination assistance component of Project GRAD Newark serves to enhance school-level capac- 
ity for program management in four ways: (1) by creating an administrative structure focused on Project GRAD imple- 
mentation; (2) by promoting school-level ownership of Project GRAD program components through staff development 
of principals and school-level vision setting and planning sessions; (3) by building awareness of and support for Project 
GRAD goals and activities among school community stakeholders (including parents); and (4) by fostering a sense of 
common purpose and cohesion across program components and a continuity of approach throughout the Project GRAD 
high school feeder pattern. 
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With primary funding from the Lucent Technologies Foundation and the Ford Founda- 
tion, Project GRAD was officially launched in the Newark Public School District in February 
1998, making Newark the second of the six school districts in the country that have adopted this 
initiative to date. A new nonprofit entity. Project GRAD Newark, Inc., was created to be the fis- 
cal agent and program manager for the initiative. The Board of Project GRAD Newark, Inc., in- 
cludes the Newark Public Schools State Superintendent and other district officials; Newark civic, 
community, and business leaders; a parent representative; and funders of Project GRAD Newark. 

With funding from the Ford Foundation, the Lucent Technologies Foundation, and the 
Grable Foundation, the Manpower Demonstration Research Corporation (MDRC) is undertaking 
an evaluation of Project GRAD Newark. The five-year evaluation is based on field research, sur- 
veys of and focus groups with teachers, and analysis of trends in student outcomes (based on stu- 
dent records) in Project GRAD Newark schools and in comparison schools. During the period 
covered by this report (January 1 998-December 1999), Project GRAD Newark schools began 
implementing four of the five Project GRAD components — the GRAD college scholarship 
guarantee. Consistency Management & Cooperative Discipline (CMCD), Communities in 
Schools (CIS), and Success for All (SFA) — and drew on program coordination assistance made 
available through Project GRAD Newark, Inc. Recently, the Project GRAD Newark, Inc., Board 
discussed the possible implementation of MathWings, which the Newark district proposed as the 
math component for Project GRAD Newark.^ This document reports primarily on the first year 
of Project GRAD Newark’s implementation. Later documents will report on its impacts on 
school functioning, student achievement, and other outcomes. 

The introduction to this report presents the findings in brief, background information on 
the theory of change underlying the reform, and the present framework for studying this model in 
Newark. The primary audience for this report is those involved in implementing Project GRAD 
or considering its adoption. As a consequence, the report presumes some background knowledge 
of the individual components that make up the reform."* 

Overview of the Findings in This Report 

• The partnership between Project GRAD Newark, Inc., and the Newark 
Public Schools has, within a relatively short time period, made significant 
strides toward establishing a strong foundation for the implementation of 
Project GRAD in Newark. 

Project GRAD Newark implementation activities, although not free of difficulty, have 
progressed relatively on target with the expected timeline: Three of the five Project GRAD com- 
ponents are already in place, and the fourth (SFA) began to be implemented in the fall of 1999. 

^The MathWings curriculum program was developed by the Success for All Foundation, the organization that 
designed the Success for All reading curriculum program that is a component of the standard Project GRAD model. 
To enhance MathWings implementation in Newark schools, the Success for All Foundation has agreed to create spe- 
cial lesson plan modules that align with New Jersey core curriculum content standards. 

“*Readers interested in learning more about the components of Project GRAD may consult the components’ Web 
sites: for CMCD, see http:www.coe.uh.edu/cmcd; for SFA, see http.//www.successforall.net; and for CIS, see 
http://www.cisnet.org. 
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This early success can be attributed to several factors jointly. First, Project GRAD Newark, Inc., 
has been diligent in communicating the goals and describing the components of the Project 
GRAD Newark initiative to raise awareness and support among school staff, students, and the 
community at large. Second, Newark school district staff supported the initiative, playing a key 
role in building Project GRAD into existing educational priorities.^ This commitment is evi- 
denced by the fact that, despite a recent turnover in district administration. Project GRAD activi- 
ties have continued in Newark and are even being considered for further expansion within the 
district. Third, the staff managing the various components of Project GRAD have been respon- 
sive in adjusting their service delivery plans to the schools’ needs. Finally, over the course of 
planning and implementation. Project GRAD Newark, Inc., and the district have become increas- 
ingly reflective about their strengths and areas in need of improvement. They continually monitor 
program status and outcomes in order to make informed decisions and mid-course adjustments so 
that implementation problems and issues can be resolved. 

• Project GRAD’S college scholarship guarantee has stimulated interest in 
and support for the initiative among school staff, students, parents, and 
local community institutions. 

Before teachers fully understood all of Project GRAD’s components, they were motivated 
to commit to the initiative primarily by the college scholarship guarantee. The multi-year funding 
commitment for Project GRAD Newark (anticipated to be about $15 million over five years) is 
seen by many as proof that the initiative is committed to the long haul. Indeed, this pledge of 
commitment by the initiative’s funders impressed school staff because, like staff at many other 
schools, they have become frustrated with short-lived and frequently changing reform initiatives. 
More important to many teachers, however, was the fact that Project GRAD Newark provided an 
opportunity for Newark students to receive college scholarships. Teachers and principals fre- 
quently cited the Project GRAD scholarship guarantee as the key motivating factor in their deci- 
sion to work toward making the initiative successful. 

Student and parent interest in the initiative, too, has been sparked by the college scholar- 
ship (hereafter referred to as the GRAD scholarship) offer. Among students in the Class of 2001 
at Malcolm X Shabazz High School (the first student cohort offered the GRAD scholarships), 73 
percent have signed GRAD scholar contracts stating their understanding of the requirements for 
the scholarship and their commitment to work to meet them. In addition, nearly 300 students in 
the Project GRAD Newark middle and elementary schools have, along with their parents, signed 
covenants pledging their support for the initiative. Community institutions have supported the 
GRAD scholarship guarantee as well. Three local colleges have helped design the Project GRAD 
Newark Summer Institute Program and implemented it on their campuses. The Mayor’s Office of 
Employment and Training also contributed to the Summer Institute Program by designating par- 
ticipating college campuses work sites, thus enabling GRAD scholars to earn money by attending 
the institute. 



*This may be partly due to the fact that a New Jersey state-mandated whole-school reform initiative accepted 
Project GRAD and included one of its components, Success for All, in its list of reform model options. In addition. 
Communities in Schools (also part of the Project GRAD model) has a nine-year history in Newark schools and a 
four-year history at Malcolm X Shabazz High School. 




• The implementation of Project GRAD components in Newark is either 
meeting or exceeding planned expectations. 

For implementation of the CMCD component, which its developers expected would be 
implemented in at least 50 percent of classrooms in each school by the end of their first full year 
in the program, most schools were on or ahead of schedule. Participation in CIS activities was 
substantial as well. Under the sponsorship of CIS, students received counseling, mentoring, and 
tutoring services; families participated in home visits, parent conferences, and special events; 165 
field trips and cultural activities were conducted; all students and staff (as well as some parents) 
participated in Project GRAD Day events at each school; and over 300 volunteers participated in 
the Walk for Success. SFA implementation, begun recently, is operating on schedule: Two 
rounds of the reading assessment have been completed at most schools (during the period cov- 
ered by this report); and Project GRAD Newark students in grades Pre-K-6 (including special 
education and bilingual students) are receiving SFA reading instruction daily. 

The continued development and growth of Project GRAD Newark suggests that the pro- 
gram components are well aligned with the challenges faced by inner-city school districts. Im- 
plementation findings to date also indicate that the Project GRAD model is malleable and re- 
sponsive to local needs. Although most of the key components used in the Houston model are 
now used in Newark, several custom-built refinements and adjustments in implementation have 
been made. (For example, the GRAD scholarship amount is higher in Newark than it is in Hous- 
ton owing to the higher local costs of college education.) This is in part due to lessons learned 
from Houston, but also to different school operating conditions and priorities in Newark. Thus, 
the Project GRAD Newark initiative demonstrates that the model is concrete, yet flexible enough 
to be adapted to a new location. 

• Although neither the reading nor the math component of Project GRAD 
Newark had yet been implemented in the year in which the achievement 
data reported here were collected, the program’s other components may 
have already begun to have a positive effect on academic achievement for 
some students. 

One of the central goals of Project GRAD is to raise students’ achievement test scores. 
The Newark school system was interested in implementing Project GRAD in part because of its 
history of low achievement test scores. And an encouraging result on student achievement has 
already been observed: The math test scores (on the Stanford Achievement Test) of 3"* graders in 
Project GRAD Newark schools were higher than would be expected from the past trend in those 
schools’ test scores. Given that such a break with past trends was not observed at other, similar 
schools in the Newark Public School District, there may be a link between the implementation of 
Project GRAD (specifically, of CMCD and CIS) and this improvement in achievement. Further, 
there is evidence that the reading test scores of 3"* graders at Project GRAD schools have also 
beaten the historical trend. At this early phase of Project GRAD implementation, test scores in 
other grades for which results are available have not exceeded expected levels based on past his- 
tory. 




The Project GRAD Newark initiative has taken important Hrst steps to- 
ward increasing schools’ capacity to implement the program. 
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Project GRAD Newark, Inc., resources have been invested largely in building capacity to 
implement the program at the school level by providing school-based staff with ongoing profes- 
sional development and fostering their program expertise. One-shot staff development work- 
shops have not been the norm. Instead, the delivery of professional development activities in the 
Project GRAD program has been intensive and ongoing. CMCD consultants have provided more 
than 80 hours of CMCD teacher training workshops. In addition, CMCD school-based staff de- 
velopers provide follow-up professional development services on a weekly basis. This support 
was instmmental in making a strong program start and building implementation momentum. 
Similarly, SFA program trainers have been responsive to implementation needs in the Project 
GRAD schools. These trainers help principals and school-based SFA facilitators provide the 
stmcture and processes for teachers to receive ongoing guidance and participate in reflective dia- 
logue on their practice of SFA techniques and their use of SFA materials. Both CMCD and SFA 
training activities have involved principals, noncertified school personnel, and central office dis- 
trict staff. However, to ensure that the components of Project GRAD are implemented effec- 
tively, sufficient time will need to be allocated to professional development activities on an ongo- 
ing basis. 

The CIS program has brought expertise in support services program development to the 
school level by hiring full-time CIS project directors for each of the Project GRAD Newark 
schools. CIS project directors participate in ongoing training provided by regional and national 
CIS offices. This training helped project directors conduct needs assessments and develop their 
school-level plans; project directors also participated in training on the Success for All Family 
Support Team. The CIS director at Shabazz High School has assumed GRAD scholarship man- 
ager responsibilities and, in collaboration with a Project GRAD Newark, Inc., Implementation 
Director, is working to strengthen GRAD scholar recmitment and monitoring activities. 

• Project GRAD Newark must increasingly turn to the complex tasks of 
meeting the academic support needs of youth in grades 7-12 and 
strengthening coordination across program components. 

With Project GRAD Newark still in the early implementation stage, much work remains to 
be done to ensure that students for whom the GRAD scholarship is intended are able to qualify for 
and take advantage of this offer. To quality for the scholarship, students must meet specified per- 
formance requirements (for example, have a grade point average of 2.5 or above, take more rigor- 
ous courses, and graduate from high school within four years). However, past academic achieve- 
ment in the Project GRAD Newark schools and the current academic achievement of GRAD schol- 
ars at Shabazz High School indicate that — unless academic supports are substantially improved — 
few students will qualify for the scholarship. Many GRAD scholars in Newark have a grade point 
average below the required level for scholarship eligibility,® and past test scores from Shabazz High 
School on state-mandated high school proficiency tests suggest that, without new supports, com- 
pleting this requirement for a diploma will be a challenge for many students. Recent reforms being 
implemented at the high school in addition to Project GRAD are an important step in the right di- 



Project GRAD Newark, Inc., staff have begun to help monitor and support students whose grade point average 
falls below 2.5. 
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rection. As Project GRAD Newark moves beyond its second year of implementation, it will be im- 
portant to introduce curricular reform components in the middle grades as well. 

There is also a strong need to turn communications and professional development efforts 
toward developing the operational linkages among the multiple programs that make up Project 
GRAD. Most of Project GRAD’s components existed as stand-alone programs in other schools 
prior to their integration into the Project GRAD model. Consequently, each component has its 
own staffing structure, professional development approach, identity, organizational goals, and 
operational techniques. With four Project GRAD program components now operating simultane- 
ously, the overlap between these components at the school level has created a need for stronger 
operational linkages, that is, greater program coherence. Although early steps in this direction 
have been taken, more focused efforts will be needed to strengthen communications and profes- 
sional development so as to enhance the operational relationships between Project GRAD com- 
ponents, particularly within the classroom. 

What Is the Theory of Change for Project GRAD? 

To meet its objectives of improving academic performance in elementary grades and in- 
creasing rates of on-time graduation from high school and college enrollment. Project GRAD 
employs the three operational strategies shown in Figure 1.1: (1) creating a school and classroom 
environment conducive to optimal learning, (2) implementing effective curricular reform in read- 
ing and math in the elementary grades, and (3) enhancing school-level capacity for program man- 
agement and implementation. To execute these strategies. Project GRAD draws on the five pro- 
gram components and program coordination assistance presented in Box 1 .2. 

To create a school and classroom environment conducive to optimal learning. Project 
GRAD employs three program components: the GRAD scholarship guarantee, CMCD, and CIS. 
These program components help create an optimal learning environment by increasing student 
engagement in school, raising students’ and teachers’ expectations as to students’ educational 
success, and improving classroom management, student supports, and social services. To imple- 
ment effective curricular reform in reading and math at the elementary level. Project GRAD uses 
the SFA reading program and a math program (MOVE IT Math in Houston) to reshape school 
curriculum and instruction. Finally, to enhance school-level capacity for management and im- 
plementation of the five program components. Project GRAD provides technical assistance and 
supplemental resources. Figure 1.1 also depicts the changes in school functioning and student 
outcomes that Project GRAD is intended to effect. Together, Figure 1.1 and Box 1.2 present the 
initiative’s theory of change. 

Evaluatine Proiect GRAD Newark 

This multi-year evaluation addresses three central questions: 

• How was the Project GRAD Newark initiative implemented? 

• Did student outcomes improve over time in the Project GRAD Newark 



schools? 
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Project GRAD Newark Operational Strategies and Intended Effects on 
School Functioning and Student Outcomes 
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Advanced Placement courses 



• How did the Project GRAD Newark initiative (as opposed to other changes 
and reforms affecting the Project GRAD Newark schools) contribute to any 
observed improvements in school functioning and student outcomes? 

Each of these questions is important to a different audience. The first is particularly im- 
portant to those implementing Project GRAD in Newark and to those in other districts consider- 
ing implementing it or beginning to implement it; the second is important to Project GRAD 
Newark, Inc., the Newark Public Schools, and the funders of the local program; and the third is 
important to the national funders of Project GRAD and to those seeking to establish research 
evidence of its effectiveness in improving troubled inner-city schools. Given that both Project 
GRAD Newark’s implementation and the MDRC evaluation are in their early stages, this report 
focuses primarily on implementation issues. It also describes the initial behavioral and academic 
performance conditions in Project GRAD Newark schools (that is, the baseline prior to Project 
GRAD implementation) and the planned method of tracking improvements in student outcomes. 
For a few key student outcomes, early evidence suggesting that Project GRAD is leading to im- 
provements is also reported. 

Program Implementation 

The present analysis of Project GRAD Newark’s implementation rests on an acknowl- 
edgment that program development occurs in stages. Staff must first launch the initiative and lay 
the administrative and funding foundation that enables program component implementation to 
begin. Once implementation activities commence, the next task is to integrate the program com- 
ponents into the daily lives of teachers and students rather than have each component operate 
separately. Crucial to strong implementation is the assembling of resources that will sustain and 
enhance the program over time. The final step in implementation is to institutionalize the reform, 
that is, incorporate it into the ongoing operations of the school district rather than have it be a 
temporary initiative dependent on special outside support.’ Prior research on education reform 
has found that throughout the program development process, it is important to communicate a 
clear, cohesive vision of the reform, to build support for the reform among key stakeholders, to 
create and then enhance school-level capacity to implement the reform, and to provide the re- 
sources to put the reform in place. 

The implementation research in this evaluation focuses on topics linked to the current 
stage of program implementation and examines the status of individual Project GRAD compo- 
nents. Topics addressed include progress in implementation, key issues and challenges encoun- 
tered, and the response of those implementing Project GRAD. When Project GRAD components 
have their own implementation benchmarks and monitoring systems (as is the case for CMCD 



’One important goal of the present evaluation is to create a standardized analysis process for examining Project 
GRAD program development. The MDRC team has begun discussions with stakeholders in Project GRAD Newark 
(including program component staff) to develop benchmarks of program component development that can be utilized 
in several ways. First, they can guide assessments of Project GRAD by providing a set of expectations about what 
should be accomplished as the program develops. Second, they allow implementers and observers of the model to 
identify gaps in Project GRAD Newark’s design and implementation, yielding information that can inform mid- 
course decisions about how to target program resources and organizational strategies. Finally, these benchmarks can 
serve as guides for other cities and schools as they implement Project GRAD, aiding the development of a blueprint 
for a strong program. 
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and SFA), this information is used to describe program status and to focus data collection on is- 
sues, challenges, and responses. 

Program Impact on School Functioning and Student Outcomes 

The analysis of school functioning and student outcomes in the MDRC evaluation of Pro- 
ject GRAD Newark is designed specifically to measure behavioral change over time.® The lower 
panel in Figure 1 . 1 depicts the four types of measures, or benchmarks, used in the analysis: 

• Measures of school climate and classroom instruction include the number 
of student discipline referrals, students’ time spent on academic tasks, student 
attendance, and attitudinal survey data (mostly from teachers) on the school 
environment.^ 

• Measures of academic achievement in elementary and middle grades in- 
clude the percentage of students who meet state proficiency standards in read- 
ing and math and comparisons of test scores to national norms. 

• Measures of student performance in high school include student course- 
taking, student academic achievement in high school (especially on achieve- 
ment tests and college entrance exams), and high school graduation rate. 

• Measures of long-run student outcomes include rates of college enrollment 
and graduation. 

Because Project GRAD Newark is in the early stages of implementation, it is not yet pos- 
sible to report on all the measures outlined above. Some, such as the high school graduation rate 
and college enrollment rate, will be covered only in later reports, once Project GRAD Newark 
students reach the appropriate stage. 

Topics Addressed in This Report 

The remainder of the report is divided into three chapters. Chapter 2 provides background 
information on the Project GRAD Newark schools and students, giving the reader a sense of the 
context in which the initiative is operating. Chapters 3 and 4 present early findings from the 
MDRC evaluation of Project GRAD Newark, based on data collected during the first 24 months 
of the initiative’s implementation. 



*An analysis comparing Project GRAD Newark schools with similar Newark schools that are not in Project 
GRAD on these measures will be performed, in addition to analyses of how both groups of schools change on these 
measures over time. Comparison analysis will not be performed for measures of school climate and classroom in- 
struction because data on these measures are not typically collected for schools not in Project GRAD. An example of 
the comparison between Project GRAD Newark schools and comparable schools is presented in Chapter 4 of this 
report. 

’Safety is a key dimension of school climate that the MDRC evaluation will track using indicators such as the 
numbers of student discipline referrals and student suspensions. 
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Chapter 2 

The Context for Project GRAD Newark 



Project grad’s goals made it an attractive option for the Newark Public School 
District — the largest in New Jersey, with an enrollment of about 45,000 students. Standardized 
test scores for students in the Newark public schools, discussed briefly in this chapter and in 
more detail later in the report, clearly indicate that many of Newark’s children are at risk of edu- 
cational failure. This chapter provides background information on the Newark school district and 
the Project GRAD Newark schools to characterize the context in which the program is being im- 
plemented. 

Three main themes emerge from the background analysis. First, owing to widespread 
concern about students at risk of educational failure, many education initiatives are under way in 
the Newark public schools. These initiatives range from the state’s takeover of the Newark 
school district to curricular reform and staff development to a variety of whole-school reform 
models. This emphasis on school improvement provides a foundation on which to build Project 
GRAD, but the diversity of existing initiatives also complicates efforts to distinguish or under- 
stand Project GRAD’S role in improving school functioning and student outcomes. Second, be- 
cause the Project GRAD Newark schools face many different operational and educational chal- 
lenges, program implementation is likely to vary between the schools and not to be perfect in any 
of them. Third, student achievement in the Project GRAD Newark schools needs to be improved 
substantially to match that of students in the rest of the state or even in New Jersey school dis- 
tricts with similar socioeconomic characteristics. But again, there is variation between the Project 
GRAD Newark schools, and in some subjects some of them have test scores substantially higher 
than the dispiriting overall picture would suggest. This variation in achievement has also influ- 
enced the implementation of Project GRAD. 

Policy and Program Developments in the Newark Public Schools 

In 1995, at the end of a 10-year investigation of the district, the New Jersey State De- 
partment of Education implemented a “takeover” of the Newark public schools, and Newark is 
still a state-operated district. According to the state’s procedure, the district will return to local 
control when it can demonstrate having reached specified performance levels on state tests ad- 
ministered in grades 4, 8, and 11; a minimum average daily attendance rate of 90 percent; a 
dropout rate below 10 percent; and adherence to state-defined operating procedures related to 
service delivery. Project GRAD was seen as having the potential to contribute to the attainment 
of these goals, which enhanced its appeal to the district. In 1999, the first state-appointed super- 
intendent of the district — who had decided to bring Project GRAD to Newark — resigned to 
become superintendent of another district, and the state appointed a long-time Newark educator 
to lead the school system. The new appointee decided to continue implementation of Project 
GRAD. Thus, the Project GRAD Newark initiative already has the stability and prominence to 
weather even a change in superintendents. 
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Newark is one of New Jersey’s 30 poor, inner-city, “special needs” districts affected by 
the New Jersey Supreme Court’s Abbott v. Burke decisions.' The 1998 decision, the latest in 30 
years of litigation seeking reform of school finance, requires the state to provide supplemental 
funding for education in special-needs districts (now often called Abbott districts) in order to 
achieve “substantial equity” between the special-needs districts and those with a larger property 
tax base. It also requires the state to provide supplemental programs and services and facilities 
improvement to special-needs districts so that students there receive “a thorough and efficient 
education,” as mandated by the state constitution. In a 1998 decision, the New Jersey Supreme 
Court endorsed “whole-school reform” as an approach that can enable students in special-needs 
districts to meet the state’s core curriculum content standards. The court also required each 
school in the Abbott districts to choose a “proven, effective whole school reform design,” which 
the New Jersey Department of Education has defined as combining “into a single program all of 
the individual educational practices and strategies that have been shown over the years to be the 
most effective in enabling disadvantaged students to achieve.”^ The court identified Success for 
All — a component of Project GRAD — as the preferred elementary school reform model. At 
the secondary school level, state guidelines implementing the court’s decision also identify Pro- 
ject GRAD as a possible whole-school reform model for Abbott districts. 

As part of a strategic planning effort following the state takeover of the school district, 
district staff identified priorities for educational change and began to effect that change.^ Much of 
their work focused on understanding the unfolding state core curriculum content standards, align- 
ing instruction to meet those standards, and adding new programs as needed. In science, staff de- 
velopment was strengthened, and teaching was oriented toward the “performance-based” state 
science test. In math, there was a restructuring of the topics to be covered at each grade level; a 
new textbook series was chosen; staff development was enhanced; and offerings of algebra were 
expanded. In language arts and literacy, goals were set for participation in lower-grade reading 
initiatives. Apart from these district-wide initiatives. Project GRAD Newark schools offer an ar- 
ray of curricular enrichment and extracurricular programs to their students. Thus, Project GRAD 
Newark was implemented in school settings that had a variety of educational and social service 
programs on which to build and that presented a coordination challenge. 

Project GRAD Newark Schools and Student Characteristics 

The Administrative Setting for Project GRAD Newark 

Project GRAD Newark, which is being implemented in a school feeder pattern that in- 
cludes elementary schools, middle schools, and a high school, was embedded into Newark’s ex- 
isting school management system — one not structured around school feeder patterns. The New- 



'See Abbott v. Burke, 153 N.J. 480, 710 A. 2d 450 (1998) for the most recent decision in this litigation. Ap- 
proximately 85 percent of the students in Abbott districts are children of color, and 75 percent receive free or re- 
duced-price school lunches, compared with statewide averages of approximately 40 percent and 25 percent, respec- 
tively. Although the 1998 New Jersey Supreme Court rulings pertained to 28 districts, two additional districts have 
since been designated Abbott districts. 

^New Jersey Department of Education Web site. 

^Newark Public Schools, 1999. 
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ark public schools are organized into five School Leadership Teams (SLTs), each managed by an 
assistant superintendent. Four of the teams each include a cluster of elementary and middle 
schools (defined along city ward lines), and the fifth team includes all the city’s 14 high schools.'* 
The elementary and middle school SLTs are designed to address the specific concerns of admin- 
istrators, schools, parents, and students in their ward, while the high school SLT addresses city- 
wide concerns regarding secondary schools. Thus, Project GRAD Newark needed to create new 
channels of communication across SLTs, bringing together staff who had previously worked 
largely independently of each other. 

The Students and Faculty at Project GRAD Newark Schools 

The Malcolm X Shabazz High School feeder pattern was chosen to implement Project 
GRAD partly because of the low academic performance of its students.^ Moreover, the high 
school had been part of a prior Ford Foundation initiative called the Urban Partnership Program.^ 

Table 2.1 lists the Project GRAD Newark schools in the feeder pattern and summarizes 
key characteristics of students and staff in the first year of program implementation. Four of the 
feeder schools were K-8, one was Pre-K-8, two were Pre-K-6, and one included grades 4-8.^ 
During the 1998-99 school year, student enrollment ranged from 41 1 to 1,323 in the five Pre-K-8 
and K-8 schools and from 338 to 747 in the two Pre-K-6 schools and was 432 in the grades 4-8 
school and 1,309 in the high school. In all nine schools, the student population is primarily Afri- 
can-American. At least 77 percent of the students in each elementary and middle school received 
free or reduced-price lunches (a good proxy for the proportion of low-income students), as did 61 
percent of the high school students.® In each of the schools, less than 15 percent of the students 
have limited English proficiency, and less than 12 percent are special education students. 

On an average day in the 1998-99 school year, more than 90 percent of students were pre- 
sent in the elementary and middle schools, and 77 percent were present in the high school. The 
average class size ranged from 17 to 22 in the Pre-K-8 and K-8 schools to about 20 in the Pre-K- 
6 and grades 4-8 schools and the high school. Student mobility ranged from 29 percent to 5 1 per- 
cent during the same school year.^ The student-teacher ratio ranged from 8:1 to 17:1 in the Pre- 
K-8 and K-8 schools and from 11:1 to 16:1 in the Pre-K-6 and grades 4-8 schools and the high 



‘‘The Newark Public School District consists of approximately 80 schools. 

second feeder pattern in Newark is expected to begin participating in Project GRAD in the near future. 



. ^ r . 



he dcvclopiaiCnt of tiiC Project GIvAI^ mOuCi in Llouston gicw out oi liic huru FUiinudUon Urb&n t'srtnersnip 
Program (UPP). Communities in Schools operated The Burger idng Academy at Shabazz High School as part of the 
Newark Educational Partnership prior to the beginning of Project GRAD Newark. 

^In the 1999-2000 school year, three Project GRAD Newark schools changed their grade configuration; two now 
serve grades Pre-K-5, and one serves grades 5-8. 

®See also Association for Children of New Jersey, 1999. Students at the high school level are generally less 
likely to establish eligibility for free or reduced-price lunch programs, so the lower eligibility rate in the high school 
should not be interpreted to mean that students at Shabazz are from higher-income families than students in the other 
Project GRAD schools. 

^Student mobility is defined as the percentage of all students who attended a school sometime during a school 
year and who entered or left that school during that year. For example, if 100 students attended a school at some 
point during a year, with 10 of them entering after the school year started and 20 leaving during the school year, then 
the school’s mobility rate would be 30 percent. 
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Key Characteristics of Project GRAD Newark Schools, 1998-99 
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Technology, Peshine Avenue is now called Peshine Avenue University Prep; and Miller Street is now called Miller Street Academy of Science and 
Technological Studies. 

^Percentage of all students enrolled in the school at any time in the year who entered or left during that year. 



school. The percentage of teachers with master’s degrees ranged from 19 percent to 44 percent in 
the elementary and middle schools and was 49 percent in the high school. 

Existing Enrichment and Extracurricular Programs in the Project GRAD 

Newark Schools 

Prior to the launch of Project GRAD Newark in 1998, the participating schools offered 
their students a variety of extracurricular and enrichment activities. Table 2.2 summarizes the 
types of such programs in each Project GRAD Newark school. These initiatives range from aca- 
demically oriented programs centered on literacy, math, and science; to community service- 
oriented programs focused on mentoring, tutoring, and leadership; to recreational programs in 
athletics and art.'° 

School Climate 

School climate in the Project GRAD Newark schools was assessed using responses to a 
survey conducted in January 1999 in which teachers were asked about their role in the school as 
they perceived it, the challenges they faced, the kinds of resources they could draw on, and the 
supports they received." Most teachers felt there was a high degree of trust and unity among 
their colleagues: 88 percent agreed somewhat or strongly that most of their colleagues shared 
their beliefs and values about the school’s mission, and 83 percent agreed somewhat or strongly 
that there was a great deal of cooperative effort among staff. In general, they also felt positive 
about the leadership and support provided by their principals (70 percent agreed somewhat or 
strongly that the school administration’s behavior toward staff was supportive and encouraging) 
and did not view principal turnover as a problem (only 30 percent deemed it a moderate or seri- 
ous problem). On the whole, teachers were less likely to report that they participated in making 
most of the important educational decisions that affected their school (50 percent agreed some- 
what or strongly) and that adequate resources were available to implement new classroom strate- 
gies as was expected of them (61 percent agreed somewhat or strongly). 

In addition, an assessment of school climate during the first year of implementation re- 
vealed significant differences in overall school climate across the Project GRAD Newark schools 
and between the school with the least favorable school climate and the one with the most positive 
school climate. Analysis at later stages of the project will examine whether these differences in 
school climate affect program implementation systematically. 



'“Communities in Schools coordinated some of these programs prior to the launch of Project GRAD Newark. 

"This survey was administered by MDRC in seven of the nine Project GRAD Newark schools. Dayton Street 
School was excluded because it had not yet joined Project GRAD, and Malcolm X Shabazz High School was ex- 
cluded because it had not yet begun CMCD program implementation. The teacher survey consisted of 27 questions 
covering respondents’ teaching load and assignments, classroom management, class resources, teaching strategies, 
perceptions and attitudes toward the school and teaching, and demographic and professional attributes; Most ques- 
tions had been used in previous surveys concerning school climate. At the time the survey was administered, there 
were approximately 285 teachers in the schools where it was fielded. The overall completion rate was 71 percent, 
with a range in individual schools from 52 percent to 93 percent. 
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Table 2.2 

Number of Enrichment and Extracurricular Programs in Project GRAD Newark Schools 



Program 


Avon 

Avenue 


Belmont Dayton 
Runyon Street 


Louise A. 
Spencer 


Madison 

Avenue 


Miller 

Street 


Peshine 

Avenue 


William 
H. Brown 


Malcolm X 
Shabazz 


Incentive 


0 


0 


0 


2 


0 


0 


0 


0 


0 


Mentoring 


1 


1 


0 


4 


2 


3 


3 


1 


1 


Tutoring 


0 


0 


2 


2 


2 


0 


1 


2 


1 


Community service/ 
leadership development 


0 


1 


1 


6 


7 


7 


5 


3 


6 


Math 


0 


0 


0 


0 


2 


0 


2 


0 


1 


Language arts/ 
literacy/joumalism 


0 


0 


0 


0 


5 


0 


3 


1 


4 


Science/technology 


0 


0 


1 


0 


5 


0 


3 


2 


7 


Athletics 


0 


0 


0 


1 


2 


0 


2 


4 


21 


Art/music 


2 


2 


2 


1 


2 


0 


4 


1 


6 


Conflict resolution 


0 


0 


0 


2 


0 


1 


0 


1 


2 


Test preparation 


0 


0 


2 


1 


0 


2 


2 


0 


0 


Extended school day 


1 


1 


1 


2 


1 


0 


2 


1 


0 



SOURCES: These data were obtained from SLT III (1999), Data Foundation Reports, 1999-2000. Newark, NJ; 

1999 SLT III Parent Conference Program; and Malcolm X Shabazz High School (1998), Malcolm X Shabazz High 
School: School Profile, 1998-99. Newark, NJ. 

NOTES: A program may be counted more than once (i.e., in more than one row) if its focus is on more than one 
area. 

^Communities in Schools operated academies with a school-within-school structure at Spencer and Shabazz prior 
to the implementation of Project GRAD Newark. 

^Spencer also had a drug prevention program and a school -wide breakfast program. 

^Shabazz also had 15 programs in association with eight local colleges and two internship programs. 
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Existing Levels of Student Achievement 



Two types of assessments of student achievement are conducted annually in the Newark 
public schools: (1) nationally normed tests of achievement initiated by the district and (2) state- 
mandated assessments measuring attainment of specific competencies. Because the two types of 
assessments follow different achievement approaches, they will be discussed separately. How- 
ever, when the reform began, test scores in the Project GRAD Newark schools were below those 
both in New Jersey and in the nation according to both types of assessment.’^ 

Norm-Referenced Tests 



During the 1990s, the Newark public schools used the Stanford Achievement Test series 
to assess student achievement in those grades between 2"‘* grade and 10“* grade for which there is 
no state-mandated assessment. (State-mandated tests are administered in grades 4 and 8.) The 
Stanford Achievement Test measures students’ academic achievement relative to that of other 
students in the same grade who took the test nationally.’^ Such a test is often called a norm- 
referenced test. Table 2.3 shows the average reading and math test scores of students in Project 
GRAD Newark schools in the 1997-98 school year.’^ In general, students in these schools were 
not performing as well as students nationally, and the discrepancy grew as grade level increased. 
Again relative to the national distribution, the math scores of students in the Project GRAD 
Newark schools tended to be higher than their reading scores. 

Three types of test results are presented in Table 2.3. The left column in the left and right 
panels of the table shows the percentage of students in Project GRAD Newark schools whose 
reading and math scores, respectively, were in the top half of the national distribution. (This cut- 
off is often called the 50“* percentile score.) In all grades, and in both reading and math, less than 
40 percent of students in the Project GRAD Newark schools scored in the top half of the national 
distribution. In addition, the percentage of students in Project GRAD Newark schools scoring in 
the top half of the national distribution declined as grade level increased. For example, 39 percent 
of 2"'’ graders in the Project GRAD Newark schools were in the top half of the national distribu- 
tion in reading, whereas only 10 percent of 10*’’ graders were. 



In the left and right panels of the table, the middle column shows the percentage of stu- 
dents in Project GRAD Newark schools whose reading and math scores, respectively, fell in the 
bottom quarter nationally. (This cutoff is often called the 25“' percentile score.) In all grades, and 
*** lywvAA tMiKA iiiutii, UL u^oot \ji MUUC1115 111 riujc(jL iNcwarK scnoois 

scored in the bottom quarter. In grade 10, the scores were particularly disheartening: In reading, 
66 percent were in the bottom quarter of the national distribution, and in math 50 percent were. 



The right column in the left and right panels of the table shows the average reading and 
math scores, respectively, of students in the Project GRAD Newark schools — reported in per- 
centiles. For example, the 39 percentile on the grade 2 reading test indicates that the average 



'^Chapter 4 — on monitoring student outcomes — provides more details on trends in testing and scores in Newark. 
^The Stanford Achievement Test includes sections on reading, language, math, science, and social science. 
Only the first three sections are administered in the Newark public schools. 

'^This discussion uses test scores for the total reading and total math subscales. 
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Table 2.3 



Stanford Achievement Test Scores for Students in Project GRAD Newark Schools, 

1997-98, by Grade 







Reading 






Math 




Grade 


Above 50th 
Percentile 


Percentage of Students 

At or Below Percentile Rank of 

25th Percentile Average Score 


Above 50th 
Percentile 


Percentage of Students 

At or Below Percentile Rank of 

25th Percentile Average Score 


2 


39 


36 


39 


36 


42 


35 


3 


21 


54 


23 


32 


45 


33 


5 


21 


50 


27 


32 


45 


34 


6 


26 


48 


29 


35 


40 


39 


7 


16 


58 


22 


30 


43 


33 


9 


6 


67 


18 


27 


38 


34 


10 


10 


66 


21 


16 


50 


26 



SOURCE: These scores on the Stanford Achievement Test (9th ed.) were obtained from the Newark Public School 
District Office. 

NOTE: Scores were converted to percentile ranks according to Stanford Achievement Test Series: Spring Norms 
Book, 9th ed. San Antonio, TX: Harcourt Brace & Company. 
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reading test score of 2"** graders in Project GRAD Newark schools was higher than or equal to 
the scores of only 39 percent of students nationally. By implication, 61 percent of 2"** graders na- 
tionally scored above the average score of 2"** graders in the Project GRAD Newark schools. 

Although the average test scores of students in the Project GRAD Newark schools com- 
pared unfavorably with the national distribution, there was variation between schools’ scores. For 
example, in grade 2, two Project GRAD Newark schools had an average score above the 50*** 
percentile in both reading and math. Similarly, two Project GRAD Newark schools had an aver- 
age reading score for 2"** graders below the 25**’ percentile, but only one school had an average 
math score below the 25**’ percentile. The average scores of students in the rest of the Project 
GRAD Newark schools fell between the 25**’ and 50**’ percentiles in both content areas. 

State-Mandated Assessments 

In 1 996, the New Jersey State Department of Education adopted core curriculum content 
standards in seven areas: language arts/literacy, mathematics, science, social studies, world lan- 
guages, health and physical education, and visual and performing arts. To measure attainment of 
these standards, the state developed a new assessment for grade 4 that began to be administered 
in the 1998-99 school year, revised its existing assessment for grade 8 in 1998-99, and will put in 
place a new assessment for grade 1 1 in 2001-02. The new tests measure mastery of specific con- 
tent areas and are being phased in during the years following Project GRAD’s implementation. 
This section, which focuses on the context for Project GRAD Newark’s launch, presents the 8**’- 
and 1 l**’-grade scores available for pre-launch years. Scores for later years, including those for the 
new 4**’-grade test, are presented in Chapter 4.*^ 

The Early Warning Test (EWT), which covers reading, math, and writing, was adminis- 
tered to 8**’ graders until the 1997-98 school year, and Figure 2.1 presents scores for students in 
the Project GRAD Newark schools who took this test in March 1998. On the reading test, 55 
percent of 8**’ graders in these schools scored as “proficient” or “advanced,” and 42 percent and 
49 percent had scores this high on the writing and mathematics tests, respectively. Beyond these 
overall scores, there is substantial variation between the Project GRAD Newark schools in EWT 
scores. Nevertheless, only one school had more than 60 percent of its students score as proficient 
or advanced in reading or math, and only two had such scores in writing. The percentage of stu- 
dents who scored as proficient or advanced in each of the EWT subjects was lower in Project 
GRAD schools than in the Newark district as a whole, in the districts in District Factor Group A 
(DFG-A), and in the state.*^ 

In grade 11, the High School Proficiency Test (HSPT) is currently used to assess skills in 
reading, math, and writing. In order to graduate with a high school diploma, students must dem- 
onstrate mastery of these topics either by passing test sections on each topic (retaking test sec- 
tions is permitted) or by completing graduation requirements through the Special Review As- 



'^Scores on these tests (which reflect the degree of attainment of core curriculum content standards) are typically 
expressed in terms of the percentage of students who demonstrate various levels of competence or proficiency (in 
grades 4 and 8) or the percentage of students who pass the test (in grade 11). 

*^In New Jersey, school districts are categorized into District Factor Groups (DFGs) based on the socioeconomic 
characteristics of their residents. The Newark public schools are part of DFG-A, the poorest group of districts. 





Figure 2.1 

Scores on the 1997-98 8**'-Grade Early Warning Test (EWT) for Project GRAD Newark Schools 

and Schools in the Rest of the District and State 
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NOTES: The 8 -Grade Early Warning Test (EWT) for the 1997-98 school year was administered in March. These data exclude students with 
limited English proficiency and special education students. 

The District Factor Group (DFG) is a measure of income, education attainment, and other demographic characteristics of district residents. It 
ranges from A in the poorest districts to I and J in the wealthiest. Newark is in DFG-A. 



sessment (SRA) process. At Shabazz High School, a substantial number of students fail one or 
more sections of the HSPT when they take the test in the fall of grade 1 1 . Figure 2.2 shows the 
percentages of Shabazz 11**’ graders taking the test in the fall of 1997 who passed the reading, 
math, and writing sections and the percentage who passed all three sections of the HSPT; for 
comparison. Figure 2.2 also shows the corresponding percentages for the Newark school district, 
DFG-A, and the state. While Shabazz students’ pass rate on the writing section is relatively high 
(48 percent), only 29 percent of them passed the reading section; 20 percent passed the math sec- 
tion; and 9 percent passed all three exams. Shabazz students’ pass rates on each of these meas- 
ures of student achievement were below those for the Newark district, DFG-A, and the state. 

Typically, more Shabazz students in a given cohort demonstrate mastery of the required 
skills over time by retaking sections of the HSPT.’* For example, in the cohort of students who 
took the test as 1 1*’’ graders in the fall of 1996, only 14 percent passed all three sections of the 
test, but 42 percent had done so by April 1998. In the cohort of students in DFG-A, 38 percent 
passed^ all three sections of the HSPT in the fall of 1996, and 71 percent had passed by April 
1998. Of the students who do not pass all three sections, some complete graduation require- 
ments by passing the SRA. Among graduates in the class of 1998 at Shabazz High School, about 
20 percent completed graduation requirements through the SRA process. 



The Special Review Assessment (SRA) provides an alternative way for students “whose knowledge and skills 
appear to be underestimated by large-scale traditional testing procedures” to demonstrate attainment of the required 
skills. By completing the SRA, seniors who have failed a section or sections of the HSPT and have met all other 
graduation requirements can demonstrate mastery of the required skills on “performance assessment tasks.” These 
tasks ^e intended to allow students to show mastery in “contexts which are familiar and related to their experi- 
ences.” See New Jersey Department of Education, 1998. 

^*^ese historical figures were obtained from the New Jersey Department of Education Web site. 

The corresponding pass rates statewide were 75 percent in the fall of 1996 and 93 percent by April 1998. 



Figure 2.2 

Scores on the 1997-98 High School Proficiency Test (HSPT) for Project GRAD Newark Schools 

and Schools in the Rest of the District and State 



c/3 









c 








o 


bO 

c 


bO 

c 




1 


cd 


iS 

’C 


•s 

cd 


C/3 






s 




m 




□ 






pjepueis yCDuapyoj j 

s|uapnis JO aSeiuaojaj 



•o ^ 
« ex 



a. 

B 

o 

•a 

V 

G 



a\ 

o\ 

43 

i 

0> 



X) 

o 

2 

B 

cd 

•O 



cd 

u 

s 

•o 

■a 



bb 

t/3 cT 

X ^ 

P =? 

c' 

C/5 d> 

U 5 



o & 

C/D X 




‘-23- 



33 



NOTES: These data are based on the fall administration of the High School Proficiency Test (HSPT). First-time test-takers and retested students are 
included, and special education students and students with limited English proficiency are excluded. 

The District Factor Group (DFG) is a measure of income, education attainment, and other demographic characteristics of district residents. It ranges 
from A in the poorest districts to I and J in the wealthiest. Newark is in DFG-A. 



Chapter 3 

Early Findings on the Implementation of Project GRAD Newark 



This chapter provides an overview of Project GRAD Newark activities during the first 24 
months of implementation (January 1 998-December 1999).' It focuses on the first evaluation 
question introduced in Chapter 1 : How was the Project GRAD Newark initiative implemented? 
The initiative’s early development is especially noteworthy given the complex nature of educa- 
tion reform generally and the particular challenges that large, inner-city school districts face 
when seeking to scale up change and improve student academic achievement in multiple schools 
at the same time. This chapter is structured around three key implementation questions: 

• What have been the major accomplishments of Project GRAD Newark thus far? 

• In what ways have communications strategies, efforts to engage key stake- 
holders, professional development, instructional leadership, and other capac- 
ity-building resources been targeted to implement Project GRAD Newark? 

• What challenges to Project GRAD Newark’s development were encountered 
during this period? 

Before these questions are addressed, it is important to note that the Project GRAD New- 
ark initiative involves several different groups of people and organizations, as illustrated in Fig- 
ure 3.1. Two of them are the Newark Public School District and the Project GRAD Newark 
schools, which are responsible for implementing the initiative. Supporting the schools in their 
implementation of the initiative are three operational entities: (1) the Project GRAD Newark, 
Inc., Board, which sets and oversees directives for the initiative, (2) the Project GRAD Newark, 
Inc., staff, who handle day-to-day project-wide management issues, and (3) the Project GRAD 
program component developers (and staff) who provide training and technical assistance to 
school staff. Project GRAD Newark, Inc., has also established community partnerships around 
Newark that help support various components of the initiative (for example, Newark-area univer- 
sities and colleges are partners in the operation of the GRAD Summer Institute Program). The 
work of Project GRAD Newark, Inc., is also linked to the Project GRAD national office.^ 



'while preliminary planning for Project GRAD Newark began prior to its official launch in February 1 998, this 
report focuses on activities conducted once implementation of the program components began. These findings 
(unless otherwise noted) are based on field research data collected by MDRC from January 1998 to December 1999. 
The following data sources were used: interviews with the Newark Public School District’s central office and School 
Leadership Team (SLT) administrators; interviews with principals; focus groups with teachers and students repre- 
senting the nine Project GRAD Newark schools (a total of 1 1 1 teachers and 42 students); interviews with guidance 
counselors in five Project GRAD Newark schools, Project GRAD Newark, Inc., staff, and program component staff 
for CIS, SFA, and CMCD; event observations; and document review. 

^The Project GRAD national office was established in the summer of 1998 for the purpose of introducing the 
Project GRAD model and providing technical assistance to cities that might adopt this initiative. The national office 
convenes Project GRAD directors periodically for cross-site planning and sharing. 
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Relationships Among Key Stakeholders in Project GRAD Newark 
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It is also important to understand Project GRAD Newark’s implementation timeline. As 
Box 3.1 illustrates, preliminary activities, such as the appointment of a Project GRAD director 
and site visits by Newark staff to Project GRAD Houston schools, were conducted in 1997. 
GRAD scholar recruitment activities and CMCD implementation began in the spring of 1998. 
The first summer institute was held in the summer of 1998. CIS activities commenced in the fol- 
lowing fall, as did the establishment of Project GRAD Newark, Inc. In 1999, the initiative con- 
tinued to grow through the first Walk for Success — a community outreach effort — and the be- 
ginning of SFA implementation. 

The Role of Project GRAD Newark. Inc. 

Project GRAD Newark, Inc., was established as the nonprofit administrator of Project 
GRAD Newark in September 1998, approximately eight months after the official launch of the 
initiative.^ Since then. Project GRAD Newark, Inc., has developed an administrative operational 
base and oversight process, which includes a cross section of school community stakeholders. It 
has focused on fostering participant engagement and capacity at the school level, primarily by 
bringing principals and school-based decision-making teams to the forefront of program planning 
and by brokering resources (especially teacher professional development) to support school-level 
implementation. In addition. Project GRAD Newark, Inc., has launched an awareness campaign 
for the initiative within schools and the community at large. Several themes have emerged from 
the early implementation activities: 

• Project GRAD Newark, Inc., is well on the way to establishing itself as a 
valued and respected external entity working in partnership with Newark 
public schools to bring about educational change. 

Working in partnership with the Newark school district. Project GRAD Newark, Inc., 
serves to: (1) create an administrative structure focused on Project GRAD implementation, (2) 
broker resources and partnerships to support the initiative, (3) foster among school community 
stakeholders awareness of and engagement in the initiative, and (4) build cohesion and unity of 
purpose among the initiative’s program components. Oversight planning for these activities is the 
responsibility of the Project GRAD Newark, Inc., Board, which is composed of the Newark Pub- 
lic Schools (NPS) State Superintendent and the Assistant Superintendent of SLT HI, the School 
Leadership Team that manages the Project GRAD schools; a parent representative; a local col- 
lege or university representative; Newark civic, community, and business leaders; and funders of 
Project GRAD Newark. Chaired by an executive officer of Lucent Technologies, the Board con- 
venes on a bimonthly basis to review the status of project activities; discuss issues related to pro- 
ject funding and outcomes, and provide recommendations and overall guidance to Project GRAD 
Newark, Inc., staff. 

Project GRAD Newark, Inc., is the fiscal agent for the Project GRAD Newark initiative. 
Its staff handles day-to-day, project- wide management issues such as budget expenditures, short- 
and long-term strategic planning, and contract negotiations with Project GRAD program compo- 
nent developers and other service-providing partners (for instance, with colleges participating in 



^Prior to the founding of Project GRAD Newark, Inc., the then-director of the Newark Educational Partnership 
(an administrator from Essex County College) served as director of the initiative. This post was assumed by an ex- 
ecutive director hired by the Project GRAD Newark, Inc., Board in November 1998. 
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Box 3.1 

Project GRAD Newark Implementation Timeline 


Fall 1997* 


Essex County College administrator is appointed Project GRAD Newark Director. 


Spring 1998^ 


Official public announcement of Project GRAD Newark is made in February. 

CMCD orientation, training, and implementation are launched in seven Project GRAD Newark 
elementary and middle schools. 

GRAD scholar recruitment activities commence at Malcolm X Shabazz High School, targeting 
the Class of 2001. 


Summer 1998 


First Project GRAD Newark summer institute is held at Essex County College. 


Fall 1998 


Project GRAD Newark, Inc., Board is established; new Executive Director is hired. 

Launch of CIS school- wide programs in the Project GRAD Newark schools. 

CMCD follow-up training for teachers and administrators is conducted. Newark-based CMCD 
staff developers are put in place; CMCD training for school paraprofessionals begins; and 
CMCD parent workshops are conducted. 

First Project GRAD Newark retreat is held in November. 


Spring 1999 


Project GRAD Newark, Inc., facilitates “visioning sessions*’ in all Project GRAD Newark 
schools. 

Malcolm X Shabazz High School and Dayton Street Elementary School receive CMCD orienta- 
tion and training. CMCD follow-up training continues in all other Project GRAD schools. 

First Project GRAD Newark newsletter and annual report are released. 

SFA Awareness sessions are held in all Project GRAD Newark schools. 

Project GRAD scholarship recruitment extends to middle and elementary schools. 

Project GRAD Principals’ Advisory Team is established. 

First series of Project GRAD Days is coordinated by CIS. 

Second Project GRAD Newark retreat is held in June. 

First Project GRAD Recognition Awards Program is held in June. 

'Newark district and school administrators visit Project GRAD Houston in November 1997; in December 
1997, Project GRAD’s founder, James Ketelsen, and CMCD’s developer, Jerome Freiberg, visit Newark 
to meet with principals at prospective Project GRAD Newark Schools. 

^Additional Newark staff visit Project GRAD Houston; teachers and some parents participate. 



O 
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(continued) 



Summer 1999 



Fall 1999^ 



O 

ERIC 



Box 3.1 (continued) 

CMCD facilitator training is conducted in Houston. 

Project GRAD Newark elementary and middle school staff receive SFA training. 

Project GRAD Newark elementary and middle school principals and teams attend SFA national 
conferences in San Francisco and Miami. 

Second Project GRAD Newark summer institute is held at three college campuses: Essex 
County College, Rutgers University, and Bloomfield College. A two-week PSAT/SAT training 
session is held in August as part of the summer institute experience. 

SFA implementation is launched. 

Full-time SFA facilitators start working in each Project GRAD Newark elementary and middle 
school. 

CMCD follow-up training continues. 

Selected teachers in each Project GRAD Newark school are released part time to serve as 
CMCD advisors. 

CIS directors coordinate the Family Support Team component of SFA. 

First Walk for Success in Newark is coordinated by CIS in November; 300 volunteers visit over 
100 homes in Newark. 

All Project GRAD Newark elementary and middle schools complete the baseline and first 
eight-week SFA assessments. 

SFA Foundation staff conduct implementation visits at each Project GRAD Newark elementary 
and middle school. 

Project GRAD Newark principals and school-based SFA facilitators attend the statewide SFA 
Leadership Conference in November. 

First SFA school facilitators' meeting is held in November. SLT III administrators and SFA 
Foundation staff convene regular meetings; facilitators from Project GRAD Newark schools 
attend. 



^Another group of Newark staff visits Project GRAD Houston; vice principals in the Project GRAD 
Newark schools participate. 
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the Summer Institute Program). Project GRAD Newark, Inc., staff members work closely with 
Project GRAD Newark stakeholders at all levels — meeting routinely with the NPS State Super- 
intendent and principals in the Project GRAD Newark schools; visiting project schools fre- 
quently; corresponding regularly with program developers and lead staff from CMCD, CIS, and 
SFA; and conducting ongoing outreach to promote awareness of the initiative among community 
agents ranging from parent organizations to local clergy and politicians. Staff work with the 
charge from the Project GRAD Newark, Inc., Board to procure services as needed for effective 
implementation. Attention in this area has been primarily focused on contracting services and 
materials from program component developers. The staff receives technical assistance in carrying 
out their work from the Project GRAD national office, located in Houston. Through ongoing 
contact with and feedback from school and district leaders (as well as service providers). Project 
GRAD Newark, Inc., staff monitor and guide the status of implementation, thus serving in a role 
that is a blend of advocate, facilitator, coordinator, and ombudsman. 

• Partnership interactions between Project GRAD Newark, Inc., and dis- 
trict staff have developed well, with the SLT III oRIce playing an espe- 
cially important role as administrative liaison for the initiative. 

SLT in provides Project GRAD Newark, Inc., staff with ongoing guidance and insight 
regarding the ways in which Project GRAD and other district initiatives can best complement 
one another within the project schools. SLT IE staff have been central to the Project GRAD 
Newark planning process, both prior to the initiative’s launch and throughout its early implemen- 
tation. For example, they have been integrally involved in coordinating visits by Newark school 
staff (especially principals and, more recently, vice principals) to the Project GRAD Houston 
schools to build administrative support for and understanding of the initiative. SLT IE staff help 
coordinate and attend nearly all staff development workshops related to Project GRAD compo- 
nents. Most recently, the SLT IE office has led early discussions with principals and the Execu- 
tive Director of Project GRAD Newark, Inc., on establishing articulation strategies to strengthen 
K-12 alignment across the Project GRAD Newark schools. 

SLT IE also serves as a conduit for information and paperwork exchange between the 
Project GRAD Newark elementary and middle schools. Project GRAD Newark, Inc., and other 
partners in the initiative, such as program component staff and the MDRC evaluation team.** A 
special assistant in the SLT IE office has been designated to handle scheduling and information 
requests regarding the project schools. This liaison has helped manage and streamline the initia- 
tive’s administrative tasks by centralizing scheduling processes involved in the multiple planning 
meetings, site visits, and other special events that are required of Project GRAD Newark schools. 
For example, CMCD program staff work with the SLT IE office to ensure that follow-up teacher 
training workshops are coordinated with the district’s staff development schedule.^ Notifications 
regarding school site visits by CMCD and SFA program staff and MDRC evaluation teams have 



^Although this is one of their responsibilities, SLT III office staff do not discourage external partners in the ini- 
tiative from making direct contact with schools. For the most part, contacts between schools and external Project 
GRAD Newark partners consist of centralized and decentralized activities in equal parts. 

^The Newark school district allocates four days to staff development during the school year. It has been impor- 
tant to schedule Project GRAD teacher training on these days because after-school and Saturday workshops for Pro- 
ject GRAD have not led to high teacher participation rates. 





been coordinated through the SLT in office as well. In addition to serving as a central filter for 
Project GRAD Newark-related requests and information directed at the schools, SLT HI recently 
assumed the role of clearinghouse for technical assistance and schools’ requests for resources 
from Project GRAD Newark, Inc. 

• With a focus on supporting principal leadership and school-level owner- 
ship of the initiative, Project GRAD Newark, Inc., has fostered relation- 
ships with schools by planning forums and collegial support activities. 

The Project GRAD Newark, Inc., Executive Director has established a direct working re- 
lationship with Project GRAD Newark principals. During the initial months of the initiative, 
principals played a relatively passive role in project-wide plans because there was no formal 
mechanism specifically for this purpose. At that time, the only forums where principals could 
come together to discuss the initiative were SLT HI meetings and CMCD principal workshops, 
neither of which were designed to convene all Project GRAD Newark principals or to address 
broad issues concerning the initiative. Principals’ roles in strategic planning for the initiative 
grew significantly when Project GRAD Newark, Inc., established the Principals’ Advisory Team. 
This team, which is composed of principals from all nine Project GRAD Newark schools, serves 
as the central planning and communications link for principals to share ideas, brainstorm, raise 
concerns, and provide recommendations on any and all aspects of the initiative. The advisory 
team is the district’s first forum for principals serving in K-12 schools to plan and work together 
on a focused reform effort. Principals generally have a say in the agenda and determine who are 
the invited guests. The Project GRAD Newark, Inc., Executive Director serves as a sounding 
board for issues raised and follows up on resolutions proposed. 

Project GRAD Newark, Inc., promotes school-level engagement in decision-making con- 
cerning the initiative beyond its work with principals alone. In November 1998, it conducted the 
first Project GRAD Newark retreat, during which School Management Team (SMT) representa- 
tives^ engaged in brainstorming and planning sessions alongside Project GRAD Newark, Inc., 
Board members. A recurring theme throughout the two-day retreat was the multi-year funding 
pledged to Project GRAD Newark, which helped assure school staff that the initiative would be 
in place over the long haul. As discussed in Chapter 1, this pledge impressed school staff because 
(like staff at many schools) they were frustrated with short-lived and frequently changing reform 
initiatives in their schools. 

SMTs attended the retreat to incorporate Project GRAD objectives into their schools’ 
five-year strategic plans and identify short-term priorities to address specific implementation 
challenges. The Project GRAD Newark, Inc., Executive Director followed up the retreat with a 
round of “visioning sessions” with the SMT at each school. The purpose of these half-day ses- 
sions was manifold: (1) to engage key school-level decision makers in a more thorough review of 
the initiative’s goals and objectives, (2) to respond to SMT members’ questions about the initia- 
tive, and (3) to continue school-level planning that focused on integrating Project GRAD proc- 
esses into SMTs’ existing strategic plan for school improvement. 



*SMTs are the state-regulated decision-making authority at the school level; their membership is composed of 
the school’s principal, teachers, other staff members, parents, and students. 
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To reinforce school-level engagement, an incentive program was developed to reward the 
efforts of school staff. Launched in the spring of 1999, the goal of the Project GRAD Newark 
Recognition Program is to assess and acknowledge exemplary practice of program implementa- 
tion and planning activities related to Project GRAD Newark.^ Because awarding incentives at 
the individual level was not common in the district, the orientation and selection processes for 
this incentive program are being refined. 

• Because Project GRAD Newark, Inc., operates largely outside the institu- 
tional chain of conunand and school bureaucracy, it is usually able to 
quickly overcome barriers to implementation that schools identify. 

Project GRAD Newark, Inc., directly or indirectly funds the bulk of the teacher training 
services, program materials, and staff resources required to implement CMCD, CIS, and the 
GRAD scholarship guarantee. For SFA, it has supplemented the school-district-funded program 
by securing additional training and materials.® Where necessary. Project GRAD Newark, Inc., 
has also hired personnel to work as “adjunct staff’ at Project GRAD Newark schools, assisting in 
the coordination and implementation of Project GRAD activities.^ In addition, it has produced an 
array of public relations materials — including a 10-minute video on the initiative in Newark — 
which have been distributed to schools to promote awareness of the goals and components of 
Project GRAD Newark.'® Such reinforcements of the Project GRAD Newark initiative’s goals, 
components, and participating schools have helped convey the “big-picture” issues that are em- 
bedded in this reform (that is, the feeder pattern approach, K-12 services leading to college en- 
rollment, and multi-intervention connections). 

Services and materials procured by Project GRAD Newark, Inc., have typically been dis- 
tributed in a standardized way across the project schools. More recently, however, a “wish-list” 



^Project GRAD Newark, Inc., established criteria for three recognition categories: (1) the GRAD Team Award 
for schools that incorporated Project GRAD into their vision for school change, identified priorities for the initiative 
in their strategic plan, and engaged a cross section of school community members in Project GRAD activities: (2) the 
GRAD Master Teacher Award for individual teachers who demonstrated mastery of and exceeded expectations for 
Project GRAD program component implementation; and (3) the GRAD Leadership Award to individual principals 
who created an environment that fostered school-level awareness, planning, and implementation of Project GRAD 
program components. The announcement of award recipients was made at the first Project GRAD Newark retreat, 
which was held in June 1999 in Atlantic City. 

®SFA implementation is financed largely by local school budgets. Project GRAD Newark, Inc., has provided 
supplemental resources to enhance SFA activities. For example, it provided a grant writer to help Project GRAD 
Newark schools raise approximately $400,000 from the Comprehensive School Reform Demonstration Program 
sponsored by the New Jersey Department of Education. Principals reported that this grant money was used to in- 
crease funding for their SFA programs. 

’Project GRAD Newark, Inc., hired an Implementation Director to help manage GRAD scholar activities at the 
high school because, unlike other Project GRAD program components, the scholarship component is not associated 
with a lead agency that can provide operational technical assistance. To enhance SFA tutoring services. Project 
GRAD Newark, Inc., has also agreed to fund a number of permanent substitute teaching positions in the Project 
GRAD Newark schools. 

'“other examples include large plaques featuring the Project GRAD Newark logo and statements of objective 
such as “Students are College Bound,” which are displayed in prominent locations throughout participating schools. 
Project GRAD Newark, Inc., has also purchased Project GRAD Newark T-shirts, which have been distributed to 
students, school staff, parents, and volunteers. 
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process has been established that allows schools to identify implementation needs specific to 
their operational context. Requests for additional materials have ranged from SFA classroom 
supplies to photocopy machines to computers and e-meiil software. In cases where a particular 
item appears on several schools’ wish lists, suggesting a common need, Project GRAD Newark, 
Inc., negotiates with school principals as a group to address this issue. 

• Project GRAD Newark, Inc., has been diligent in communicating the 
goals and components of Project GRAD to school staff and more recently 
has begun to promote dialogue with staff of the Project GRAD compo- 
nents on issues of overall program coherence. 

The initial public announcement of Project GRAD Newark, which kicked off the initia- 
tive, was made at Malcolm X Shabazz High School during a student assembly and press confer- 
ence held on February 17, 1998. Although this announcement attracted considerable media cov- 
erage, a project-wide strategy to promote awareness of the Project GRAD model was not strongly 
in place when Project GRAD Newark was launched. Consequently, the information for teachers, 
students, and the local community about the initiative was uneven in clarity and comprehensive- 
ness. For example, student and parent meetings specifically concerning Project GRAD Newark 
were conducted only at the high school and focused solely on the GRAD scholarship offer. The 
only formal project-wide forum for elementary and middle school teachers at the launch of the 
initiative was an orientation on CMCD; little information on the other four components of Pro- 
ject GRAD was presented.” For the high school teachers, there was no advance orientation. 
Teachers reported that they first learned of the initiative the day it was announced at the student 
assembly or through media coverage of the program.’^ Similarly, CIS activities sponsored by 
Project GRAD Newark commenced in the schools with almost no prior formal introduction for 
school staff.’^ As a result, during the first several months of implementation (that is, February 
1998-November 1998), there appeared to be minimal understanding among teachers of the scope 
of the Project GRAD model’s components. As a result, many teachers perceived the reform effort 
as consisting of only CMCD and the offer of college scholarships at Shabazz. It is important to 
note, however, that just knowing about the GRAD scholarship guarantee sparked teachers’ inter- 
est. In fact, teachers often cited the GRAD scholarship as their primary motivation for wanting to 
be a part of the initiative, echoing the sentiments of a teacher who said, “We are doing this for 
the kids! We are doing this for scholarships for the kids.” While information on the GRAD 
scholarship and CMCD orientation appeared to be sufficient to win teacher support for Project 
GRAD Newark, limiting the early communications strategy may have undermined school com- 



''Communications introducing Project GRAD were made available at individual schools to varying degrees. 

'^High school teachers did not receive an orientation on CMCD during the first year of Project GRAD implementa- 
tion, in conformity with the established policies and procedures of CMCD, which normally focuses on elementary and 
middle schools prior to high school implementation. However, because this aspect of implementation was not explained 
to the high school teachers early on, they tended to express greater detachment from Project GRAD implementation than 
did teachers in other Project GRAD Newark schools. 

'^This is not to say that a CIS orientation process was not desired. In the spring of 1998, the CIS of New Jersey 
Executive Director made several attempts to schedule a principals’ meeting to discuss CIS. However, owing to de- 
lays in CIS program funding, the first opportunity to meet with principals came the following fall, when CIS imple- 
mentation was to commence. Principals and CIS project directors described the introduction process as awkward. 
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munity stakeholders’ ability to develop a solid understanding of Project GRAD Newark’s objec- 
tives and components. 

A number of vehicles for increasing awareness and understanding of Project GRAD 
Newark’s components were established during the 1998-99 school year. For school staff, the 
availability of information sources on the initiative increased substantially as efforts to build 
awareness grew. For example, at the first CMCD teacher training session in the 1998-99 school 
year, teachers were shown a video on Project GRAD Houston, which, for the first time since the 
Newark initiative’s inception, clearly delineated each of the program components encompassed 
in this model.* *'* In addition, materials describing the initiative (for example, a Project GRAD 
Newark fact sheet, newsletter, and annual report) were distributed to the broader school staff 
membership. More recently, an electronic network — the Newark Houston Connection — has 
been established to allow 30 teachers from Project GRAD Newark schools and 30 teachers from 
Project GRAD Houston schools to discuss their experiences concerning the initiative on a regular 
basis. The network provides computer laptops and computer training to two or three teachers in 
each Project GRAD Newark school; these teachers, in turn, commit to maintaining regular dia- 
logue with other network participants and to sharing ideas and lessons that emerge from these 
discussions with the broader staff in their schools.’^ 

Increased attention has also been devoted to building parent awareness of Project GRAD 
Newark. At each of the Project GRAD Newark schools, at least one parent awareness meeting 
was conducted in which representatives from the board and staff of Project GRAD Newark, Inc., 
the NPS central office, the host school, and each of the Project GRAD program components pre- 
sented information and responded to parents’ questions about the initiative. Parent workshops on 
CMCD were also conducted at each school during the fall of 1998 and the spring of 1999. In the 
fall of 1999, parent outreach activities were expanded further when the initiative launched its 
first annual Walk for Success, in which school staff and volunteers visited parents at their homes 
to explain the initiative and distribute Project GRAD materials (see Box 3.2).*^ Through its re- 
newed emphasis on parent outreach, the SLT m office has also helped promote awareness of 
Project GRAD Newark by highlighting the initiative at a number of parent involvement events: 
the SLT in Parent Conference (held in December 1999), monthly Direct Link parent meetings,*^ 
and parent “chats” conducted by the SLT HI Assistant Superintendent in the fall of 1999. Aware- 
ness of the initiative among students in the Project GRAD Newark schools (particularly in the 
elementary and middle grades) probably grew the most during the 1998-99 school year — thanks 
to Project GRAD Day. Under the coordination of CIS staff, each school prepared a number of 
college awareness activities that culminated in this day-long, school-wide event, which used in- 
formation on Project GRAD as the backdrop for a celebration of students’ current talents and po- 
tential achievements after high school and college. 



'■’Project GRAD Newark, Inc., subsequently produced a video highlighting Newark implementation activities, 
which is now shown at most Project GRAD public awareness meetings. 

'^This initiative is sponsored by the Project GRAD national office. Participating teachers volunteer to participate 
in the network after being nominated by their building principal. 

*Walk for Success activities are described in greater detail later in this chapter. 

Direct Link meetings serve to inform parents of district policies and education initiatives being implemented in 
SLT III schools. 
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Box 3.2 



Project GRAD Newark Walk for Success 

Developing a Strong Working Relationship with Parents of Mutual Respect. 

Trust, and Caring 

On a crisp, bright Saturday in November 1999, over 300 volunteers from the Newark public 
schools, the surrounding community, and area businesses participated in the first Project GRAD 
Newark Walk for Success. The Walk for Success served to encourage families to become more in- 
volved with their children’s schools, identily needs that parents feel the Project GRAD Newark initia- 
tive can address, and recruit more Project GRAD scholars. 

The volunteers, among them a “dream team” composed of the NPS Superintendent, the Project 
GRAD Newark, Inc., Board Chairman, and a GRAD scholar from Malcolm X Shabazz High School, 
conducted a door-to-door campaign to introduce parents of students in Project GRAD Newark 
schools to the opportunities offered by the program. The mission of the event was to “develop with 
parents a strong working relationship of mutual respect, trust, and caring.” 

The nine Project GRAD Newark schools served as command centers for the day’s activities, 
providing orientation to the volunteers, deploying volunteer teams into the surrounding neighbor- 
hood, and serving as information points for parents, students, and others who wanted to come to the 
school to discuss the initiative. After visiting over 100 homes and collecting 275 signed covenants 
and 30 signed GRAD scholar contracts, volunteers convened at Malcolm X Shabazz High School to 
celebrate their rewarding first effort. 
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As more of the Project GRAD Newark program components come on line — most re- 
cently, SFA, which was introduced in the fall of 1999 — communications efforts are turning to 
the issue of program coherence.’® With several components of Project GRAD Newark now oper- 
ating in schools, the initiative is faced with the challenge of establishing clear and consistent 
communications to facilitate the coordination of Project GRAD components. Possible strategies 
for tackling this challenge are presented later in this chapter. 

The GRAD Colle£e Scholarship Guarantee and Summer Institute Program 

The GRAD college scholarship guarantee is often referred to as the cornerstone of Project 
GRAD Newark because it most directly represents the ultimate goal of the initiative, which is to 
increase rates of college enrollment and graduation. During the past 24 months, activities to 
stimulate students’ interest in the GRAD scholarship offer have ranged from parent-night meet- 
ings to neighborhood canvassing. The majority of Malcolm X Shabazz High School students in 
the first Project GRAD Newark cohort, the Class of 2001, have already signed GRAD scholar 
contracts. For these and future GRAD scholars at the high school level, the initiative has focused 
on enhancing the academic experience through the Summer Institute Program as well as through 
after-school support activities coordinated by CIS. However, academic achievement records for 
GRAD scholars already in high school indicate that these activities may not be enough for them 
to meet the GRAD scholarship eligibility requirements. For example, relatively few of the 
GRAD scholars at Malcolm X Shabazz High School appear to be on the path to earning a 2.5 
grade point average by the time of their scheduled graduation. 

• Significant strides have been made in GRAD scholar recruitment, but 
these activities are not yet well integrated with existing mechanisms for 
student orientation on choosing a secondary school. 

Student recruitment is a key priority for the GRAD scholarship component because — al- 
though the scholarship is offered to al\ students in Project GRAD Newark schools — each student 
and his or her parent(s) are asked to sign a contract in order to be officially bound to the program’s 
scholarship eligibility requirements (see Box 3.3). Initial recruitment activities centered on Mal- 
colm X Shabazz 9* graders. For the first cohort of GRAD scholars, the Class of 2001, recruitment 
began in February 1998 with a series of six orientation sessions designed to inform 9*’*-grade stu- 
dents and parents about the scholarships. However, the benefits of the scholarship offer were, in the 
words of one Shabazz administrator, “harder to promote” because recruitment started too late to 
excite students’ and parents’ interest and aspirations effectively. Since that time, GRAD scholar 
awareness activities have expanded consider^ly. While recruitment activities continue at the high 
school (focusing primarily on 9* and 10* graders), elementary and middle grade students are now 
being recruited as well, starting as early as kindergarten. Recruitment activities are directed at get- 
ting students in grades 8, 9, and 10 to sign GRAD scholar contracts and students in grades K-6 to 
sign a Project GRAD “covenant.” According to December 1998 estimates, 73 percent of students in 
the Class of 2001, 56 percent of students in the Class of 2002, and 14 percent of students in the 
Class of 2003 have signed GRAD scholar contracts. Approximately 275 elementary and middle 
school students have signed covenant agreements. 



’®The Executive Director of Project GRAD Newark, Inc., has supported this process by convening Project 
GRAD program component lead staff in discussions related to program coordination. 
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Box 3.3 



Project GRAD Scholarship Contract 



As part of their official commitinent to the program, students, parents, and the Ex- 
ecutive Director of Project GRAD Newark, Inc., sign a contract called the Project GRAD 
Newark Guarantee. The current contract guarantees a $6,000 college scholarship (annual 
payments of $1,000 during the freshman and sophomore years and $2,000 during the junior 
and senior years) to GRAD scholars who meet the following requirements: 

• Remain enrolled at Malcolm X Shabazz High School until graduation 

• Graduate within four years of their enrollment 

• Have a cumulative GPA of at least 2.5 at the time of graduation 

• Participate in two Project GRAD summer institutes (or a pre-approved equiva- 
lent program) 

• Complete all college preparatory academic requirements as outlined by the 
school district and the state of New Jersey' 

• Take and pass all tests required by the district and state 

The GRAD scholarship contract has gone through several iterations since it was first 
drafted in the spring of 1998, partly because of Project GRAD Newark, Inc.’s, responsiveness 
to Newark parents* and teachers* concerns about its fairness to students. For example, re- 
quirements to do community service and to take both the PSAT and SAT are not included in 
the most recent version of the contract. 

Once enrolled in college, GRAD scholars must maintain a minimum GPA of 2.0 in or- 
der to retain their scholarship. The scholarship is available for up to six years after high school 
graduation. 

^Although no formal district or state document stipulates college preparatory requirements. New 
Jersey’s general high school graduation requirements were scheduled to be revised in March 2000. 
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Students were compelled to sign up for the GRAD scholarship program through a variety 
of channels, primarily student and parent orientation sessions. These sessions, generally con- 
ducted in the evening, are held at least once a year at each of the Project GRAD Newark elemen- 
tary and middle schools and more frequently at the high school.’® Project GRAD presentations in 
9'*’ and 10*’’ grade homerooms, in which students receive an overview of scholarship benefits and 
requirements and are given GRAD scholar contracts to take home for their parents to sign, are 
also conducted at several points during the year. In the spring of 1999, Project GRAD Days be- 
came another vehicle for GRAD scholarship recruitment, particularly for S'** grade students at the 
elementary and middle schools who, as part of the celebration, participated in a special orienta- 
tion session in which the contract was reviewed and sent home for students to return. More re- 
cently (November 1999), recruitment activities were “taken to the streets” when Project GRAD 
Newark implemented its first annual Walk for Success, which encouraged parents and students to 
participate in the scholarship program. 

While widening the array of GRAD scholar awareness and recruitment activities has 
yielded positive results, such activities operate fairly independently of Project GRAD Newark 
schools’ existing infrastructure for providing students with information and guidance related to 
high school options. Elementary and middle school guidance counselors report that they spend a 
considerable amount of time working with S'** graders on high school selection processes; coun- 
selors also serve as contacts for parents who want to learn about high school programs and op- 
tions. However, GRAD scholar recruitment efforts have yet to engage systematically the in- 
volvement, resources, or expertise of guidance staff in a way that might help advance recruitment 
activities. 



• The Project GRAD Newark Summer Institute Program has been success- 
ful in giving GRAD scholars a college campus learning experience that 
focuses on academic enrichment. 

The Summer Institute Program operates for four weeks in July with the intent of provid- 
ing GRAD scholars with academic enrichment, college campus exposure, and a small-group 
learning environment that builds camaraderie among participants. Two institute programs were 
implemented during the period covered by this report. The first was completed by 126 GRAD 
scholars in the summer of 1998; the second was completed by 196 scholars in the summer of 
1999. 



Summer institute curricula were designed by participating college faculty and included 
courses in subjects ranging from English, biology, mathematics, and the humanities to environ- 
mental law, geology, astronomy, and media literacy. A variety of teaching approaches were used 



* ^Participants in these sessions receive an overview of the Project GRAD Newark initiative’s goals and compo- 
nents (including a video presentation) and a more thorough review of the GRAD scholarship requirements, and 
GRAD scholar contracts are distributed for them to sign. 

because the position of head of guidance at the high school was vacant during the 1998-99 school year, no in- 
terviews with guidance staff at the high school were conducted in that school year. However, among the five guid- 
ance counselors interviewed, there was a general sense that handling issues at Malcolm X Shabazz High School 
(such as its reputation for fighting and for a low-level curriculum) and students* and parents* desire for high school 
magnet programs are key challenges to GRAD scholar recruitment. 
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in summer institute classes, including small cooperative groups, interactive projects, theme-based 
or interdisciplinary instruction, technology, large-group lectures, seminars, and field trips. Sum- 
mer institute faculty reported that hands-on instruction, technology-based learning, and academic 
field trips were the most effective strategies for engaging students’ interest; traditional, 90- 
minute college lecture formats were least effective. These experiences have helped institute fac- 
ulty acknowledge the need to adjust their ideas about effective teaching practice in order to better 
engage secondary students in learning. 

• Local institutions of higher education were key partners in developing 
and operating the Summer Institute Program; however, more involve- 
ment on the part of Malcolm X Shabazz High School staff and parents is 
desirable. 

The Project GRAD Newark Summer Institute Program builds upon existing Newark 
Educational Partnership (NEP) activities in which staff at Malcolm X Shabazz High School and 
local college and university staff work together to develop articulation agreements that enhance 
students’ transition from secondary to post-secondary learning environments.^’ With funding and 
administrative service support from Project GRAD Newark, Inc., lead participants in the NEP lay 
the operational foundations for the Summer Institute Program by providing planning time, staff, 
curricula, facility space, and student participation incentives. 

The 1998 Summer Institute Program was staffed by faculty members at Essex County 
College and Rutgers University; all classes were conducted at the Essex County College campus. 
By the institute’s second year, its classes were being held at two additional campuses — those of 
Rutgers University and Bloomfield College.^^ An administrator from each college served as di- 
rector of the respective institute; the directors met regularly with the Project GRAD Newark, Inc., 
Executive Director to plan administrative logistics related to application, registration, and record- 
keeping procedures.^^ Also, the Mayor’s Office of Employment and Training donated resources 
to the program by designating each participating campus a work site for the city’s Summer Youth 
Employment and Training Program (SYETP), which enabled students to use the summer insti- 
tute as their work site and earn money for attending the institute, an important incentive for 
GRAD scholars.^'’ 

Summer institute faculty reported the need for greater involvement from two important 
parties — GRAD scholars’ parents and Malcolm X Shabazz High School staff. Parent participa- 



^'NEP was established in 1991 as part of the Ford Foundation-sponsored Urban Partnership Program. 

“Each campus offered both full- and half-day class schedules to accommodate students who were required to at- 
tend NPS summer school during part of the day, but summer institute staff recommended that this be avoided next 
year to alleviate administrative and logistical problems with scheduling. 

“a Project GRAD Summer Institute Advisory Committee was established in January 1999 to plan and coordi- 
nate the Summer Institute Program. Comprising the summer institute directors from each host college campus, mem- 
bers of the faculty at each campus, the principal of Malcolm X Shabazz High School, and Project GRAD Newark, 
Inc., staff, the committee has met on an almost monthly basis to discuss operational objectives for the summer insti- 
tutes, establish partnership roles, design curricula, and identify accomplishments in and address challenges to pro- 
gram implementation. The committee planned the summer institutes held in 2000. 

^he logistics of incorporating SYETP required more paperwork than faculty anticipated; plans are being made 
to develop a more efficient procedure for paying students next year. 
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tion in and contact with the Summer Institute Program was considered low. Faculty attributed it 
partly to the fact that, aside from the closing ceremonies held at the end of each summer, the pro- 
gram had not created opportunities for parent participation. Similarly, faculty members felt that 
teaching and learning experiences for GRAD scholars could be enhanced through the develop- 
ment of connections between curricula in the high school and the curriculum used in institute 
classes. Efforts are under way to include department chairs from the high school in the planning 
process for the 2000 Summer Institute Program in order to encourage cross-faculty curriculum 
design and to align institute activities with New Jersey core curriculum content standards. There 
have also been discussions about including summer institute faculty in CMCD training. CMCD 
not only teaches classroom management strategies (aimed at curtailing disciplinary problems), 
but includes techniques to increase the number and quality of parent communications. ^ 

• In addition to efforts currently in place to prepare GRAD scholars to 
meet the GRAD scholarship eligibility requirements, more supports are 
needed to strengthen students’ academic skills development. 

Eligibility requirements for the GRAD scholarships are reviewed at nearly every Project 
GRAD Newark event and are carefully spelled out in the GRAD scholar contract. Many of the 
academic requirements outlined in the contract are very similar to those for eligibility for a high 
school diploma mandated by the state or individual districts (such as passing the state-mandated 
High School Proficiency Test and earning at least 1 10 academic credits). Others, such as the 2.5 
minimum grade point average and completion of two summer institutes (or a pre-approved 
equivalent program) are unique to the GRAD scholar agreement. 

Support systems for GRAD scholars at the high school began to develop during the 1998- 
99 school year when the school district provided funds to contract the services of a “scholarship 
manager.”^^ The scholarship manager is responsible for conducting GRAD scholar awareness 
and recruitment activities at the high school, developing processes to monitor and support the 
academic progress of GRAD scholars, and coordinating the application and registration process 
for the Summer Institute Program.^® The scholarship manager is assisted in these activities by an 
Implementation Director from the Project GRAD Newark, Inc., office. The scholarship manager 
works out of the high school’s GRAD scholar office, which serves as the central location where 
students, teachers, and parents can obtain information on GRAD scholarship activities. Current 
and prospective GRAD scholars are encouraged to visit the office to discuss the eligibility re- 



“CMCD program staff reported that over 900 parent contacts were conducted by Malcolm X Shabazz faculty 
(using CMCD materials) from September 1999 to January 2000. 

^^In addition, the GRAD scholarship requires students to complete three years of college preparatory math 
courses, whereas the state requires three years of math, which need not be at the college preparatory level. 

district-hired consultant (based at the high school) served as the Project GRAD Newark scholarship manager 
during the 1998-99 school year. The position remained vacant thereafter — until the CIS director at Malcolm X 
Shabazz High School was hired as the scholarship manager in February 2000. 

^*The Project GRAD Houston scholarship manager has several critical responsibilities: (1) leading recruitment 
of GRAD scholars and college planning activities, (2) serving as primary liaison to parents of GRAD scholars, (3) 
keeping track of GRAD scholars* progress toward meeting requirements, (4) developing tutoring and other academic 
and social support programs for GRAD scholars, (5) promoting awareness of Project GRAD within the school and 
surrounding community, and (6) developing and maintaining college and business partnerships to support GRAD 
scholars. 
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quirements, their progress toward meeting those requirements, and the registration procedures for 
enrolling in the summer institute. 



A key focus of the scholarship manager during the 1998-99 school year was to create a 
database system for monitoring the program and the academic status of GRAD scholars. The 
GRAD database houses information such as students’ grade point average (GPA), high school 
credits earned, number of GRAD summer institutes completed, and PSAT and SAT scores. Set- 
ting up the database turned into a time-consuming process because many of the variables relevant 
for GRAD scholarship eligibility were not included in the school district’s existing student re- 
cords system. Guidance counselors at Malcolm X Shabazz High School are currently working 
with Project GRAD Newark, Inc., staff to incorporate the GRAD scholar database into the 
school’s existing electronic student records. This integration of record-keeping systems may 
prove to be an important step toward institutionalization of the Project GRAD Newark initiative. 



Database records show that the majority of current GRAD scholars at Malcolm X Shabazz 
High School are not on track to reach the minimum performance threshold. As of December 1999, 
database records indicated that among GRAD scholars, 10 percent in the Class of 2001, 12 percent 
in the Class of 2002, and 14 percent in the Class of 2003 have GPAs of at least 2.5. For GRAD 
scholars in the Class of 2001, this finding warrants special concern because, to qualify for the 
GRAD scholarship, they must raise their cumulative GPAs under tight time constraints, that is, be- 
fore the end of their years of GRAD scholarship eligibility. Project GRAD Newark, Inc., staff and 
school administrators are discussing options that would allow these students to enroll in an acceler- 
ated program of study so that they could complete more courses than they otherwise would in the 
same period of time. Project GRAD Newark, Inc., has also identified the need to give GRAD 
scholars more assistance in understanding how to calculate and monitor their GPAs so that they re- 
alize how their grades affect their eligibility for the GRAD scholarship. It should be noted, how- 
ever, that an intervention is being developed that would provide mentoring and tutoring to GRAD 
scholars in the 1 1*** grade whose cumulative GPAs are below 2.5. 



GRAD scholars must also pass all three sections of the New Jersey High School Profi- 
ciency Test (HSPT) or — if they do not demonstrate their mastery of the subject matter covered 
by the test — successfully meet the requirements of the Special Review Assessment (SRA), 



which provides an alternative route to graduation.^® This is a requirement for high school gradua- 
tion and, therefore, for the GRAD scholarship. Thus far, however. Project GRAD Newark’s ef- 
forts have not focused on helping students prepare for the HSPT, despite the fact that in recent 
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passed all three sections of the test. The grade 1 1 test is being revised to align more closely 
with New Jersey core curriculum content standards, and a new test will be administered for the 



^^Because GPA is not a standard performance measure on district report cards, most students are familiar with 
neither the term nor how it is calculated. The scholarship manager designed a lesson plan for teachers to use in help- 
ing students calculate their GPA, but it was not systematically used. 

^^The state tests and the SRA are discussed in more detail later in this chapter and in Chapter 4. 

^*The HSPT has three sections — one each on math, reading, and writing. In order to receive a New Jersey high 
school diploma, students must pass all three sections by the time of graduation or complete the SRA. As mentioned 
earlier, students can take each section of the test up to four times. Support activities are included in the summer insti- 
tutes to help strengthen math and language arts and to help students prepare for the PSAT and SAT. 
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first time in 2001-02. This new test is expected to be more demanding, intensifying the challenge 
Project GRAD Newark faces in getting students to meet it. 

Three program components of Project GRAD Newark have been put into place thus far to 
increase the likelihood that students meet the Project GRAD eligibility requirements (and go on 
to graduate from college): CMCD, CIS, and SFA.^^ While two of these programs (CMCD and 
CIS) are being implemented in all the Project GRAD Newark schools, SFA implementation fo- 
cuses on students in grades Pre-K-6. For current GRAD scholars at Malcolm X Shabazz High 
School (and, possibly, for the class cohorts immediately following) who will not reap the benefits 
of Project GRAD’s reading and math program components, the need for additional academic 
supports is immediate. 

The recent adoption of the Talent Development High School model at Malcolm X Sha- 
bazz High School may help fill this academic and operational gap. Launched in September 1 999, 
this reform model is being used to restructure the high school’s instructional program (by estab- 
lishing a new core curriculum focused on college-preparatory coursework) and organizational 
operations (by grouping teachers and students into learning academies). Later in this chapter, is- 
sues related to Project GRAD’s potential role in building a stronger academic program at the 
middle and high school grades are discussed. 

Consistency Management & Cooperative Discipline (CMCD) 

CMCD implementation, which began in the Project GRAD Newark schools in March 
1998,^^ has been supported through ongoing training workshops, CMCD materials, and technical 
assistance as needed. CMCD training has targeted a broad range of school staff, including teach- 
ers, administrators, and paraprofessionals. CMCD operations are already well established in the 
Project GRAD Newark schools; on average, schools were meeting (and, in most cases, exceed- 
ing) the expectations of program staff for in-classroom CMCD implementation. CMCD imple- 
mentation has also led principals to establish a new discipline referral process in all Project 
GRAD Newark schools. In keeping with the usual pattern of CMCD program development, staff 
have identified critical common areas in the school (the cafeteria, playground, auditorium, and 
hallways) as the next target for strengthening program implementation. 

• The accessibility of technical assistance provided by CMCD program 
staff, as well as the ongoing nature of CMCD teacher training, has been 
instrumental in the implementation of CMCD. 



^^The SFA math curriculum program, MathWings, will begin to be implemented as part of the Project GRAD 
initiative during the 2001-02 school year. 

^^CMCD staff conducted an orientation workshop for potential Project GRAD Newark elementary and middle 
schools in January 1998. At seven of the schools, the percentage of teachers who gave affirmative votes reached the 
CMCD standard of at least 70 percent needed to implement the program in the spring of 1998. In one elementary 
school, the standard of 70 percent was not met in 1998 but was reached in the spring of 1999. While there had been 
earlier discussion among Malcolm X Shabazz faculty members about the Project GRAD Houston visits and the 
CMCD component, the full staff received its first CMCD orientation in October 1998. At that time, 96 percent of the 
teachers voted in favor of implementing CMCD in the spring of 1999. 
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At the initial CMCD orientation, Project GRAD Newark schools were promised that 
CMCD training would not consist of one-shot staff development workshops after which staff 
would be expected to implement the program without further training or support. In fact, since 
the introductory workshops, Houston-based CMCD consultants have conducted ongoing training 
and school visits to reinforce and deepen staff understanding and implementation of the CMCD 
program. During the 1998-99 school year, 10 three-hour follow-up CMCD workshops were con- 
ducted. During the 1 999-2000 school year, five workshops for Project GRAD Newark schools in 
their second year of CMCD implementation and five for those in their first year had been held by 
the end of the period covered by this report.^'* In total, over 80 hours of CMCD training have 
been provided to Project GRAD Newark teachers since the initiative began. Initial CMCD train- 
ing was held on a Saturday in the spring of 1998 and was attended by staff members who volun- 
teered to participate. Twenty percent of the staff at the Project GRAD Newark schools attended. 
CMCD conducted three additional training sessions that spring, for a cumulative attendance rate 
of 82 percent, in order to give the majority of teachers an opportunity to receive the introductory 
training in CMCD’s philosophy and preventive strategies. 

Since this integration, teacher participation (across the Project GRAD Newark schools) 
has been consistently high: Participation rates averaged 80 percent in the 1998-99 school year 
and 88 percent through December in the 1999-2000 school year. Project GRAD Newark teachers 
generally reacted very positively to CMCD’s professional development delivery model, as the 
comments of focus group participants such as the following indicate: 

I liked that they gave us a workshop to really explain things and each workshop 
was built on the previous workshop. They would ask us for feedback and would 
give us additional ideas. . . . The big benefit was to have individuals come in 
who are using [CMCD]. It was actually teachers coming in to talk to us. . . . 

They didn’t throw it all at us at one time; they gave it to us piece by piece like 
we do with the kids. . . . The workshop was great; I got something to take. ... I 
really liked that we got hands-on materials to bring back. . . . When I left, I was 
eager to try things. . . . We did hands-on things to start off with. The workshop 
was a Saturday — we could start [implementing] on Monday! . . . They didn’t 
say, “We’ll send you ’’ They gave it to us immediately. 

CMCD program staff have devoted particular attention to providing teachers newly as- 
signed to the Project GRAD schools with introductory training in the program before they enter 
cJassrooms or participate in CMCD follow-up workshops for returning teachers : In other 
words, new teachers are not left on their own to figure out the basics of the program — they re- 
ceive the same training foundation as was provided to the rest of the school staff. This careful 
and thorough approach to teachers’ professional development has helped the program counteract 
implementation “slippage” that can result from teacher turnover. 



^At the end of the period covered by this report, seven Project GRAD Newark schools were in their second year 
of CMCD implementation, and two (the high school and Dayton Street Elementary) were in their first year. 

^^New teacher training workshops are conducted at the start of each semester. CMCD staff developers provide 
one-on-one or small-group training for teachers who are newly assigned to the building during the course of the 
school year. 
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Early in the implementation process, CMCD staff also put strategies into place to develop 
capacity within the Newark district (and particularly in schools) to sustain and renew teachers’ 
implementation of the program. Newark-based CMCD staff developers were hired to provide 
school-level coaching support to teachers. Three staff developers were working at the start of the 
1998-99 school year, and two more were hired in the following spring. All CMCD staff develop- 
ers are former Newark public school classroom teachers who were introduced to CMCD while 
teaching in Project GRAD Newark schools. Each undergoes additional training in Houston to 
prepare for the role of CMCD staff developer for Project GRAD Newark. Staff developers work 
as a team to conduct student assemblies on CMCD, provide training for newly hired and substi- 
tute teachers, post and update CMCD bulletin boards, replenish school supplies of CMCD mate- 
rials, collect discipline referral forms, and conduct CMCD parent workshops. In addition, each 
staff developer is assigned a particular set of schools to visit at least weekly in order to help 
school administrators and teachers refine CMCD implementation. CMCD staff developers meet 
individually with their assigned schools’ principals at least monthly to collaborate on school 
management topics and site-specific issues related to program implementation. Finally, CMCD 
conducts surveys with students, teachers, and the administrative staff to elicit their perceptions of 
the school. These data on school climate are compiled and shared (without identifying sources by 
name) with staff each year in efforts to identify strategies for addressing the school’s needs. 

In an effort to further strengthen school-level capacity for CMCD implementation, in May 
1999 each Project GRAD Newark school selected a group of teachers to assist the CMCD staff 
developers.^^ This additional support strategy is a regular part of the CMCD program’s evolution. 
Tj^ically, in the second year selected teachers (usually two teachers per school) are released for 
one prep period per week to serve as CMCD advisors. The advisors are expected to support 
CMCD staff developer activities, paying particular attention to working with new teachers. Col- 
lectively, the advisors provide 1 10 minutes of CMCD staff development each week. 

• By the end of their first full year using CMCD, Project GRAD Newark 
schools had generally met the program developer’s expectations, with 50 
percent or more of all classrooms in each school having implemented 
CMCD program techniques. 

Classroom observations revealed that the target for CMCD implementation in each Pro- 
ject GRAD Newark school — that is, that 50 percent or more of all classrooms would have im- 
plemented the program — was generally exceeded.^’ CMCD techniques (too numerous to list 
here) reflect the program’s five operational themes: prevention (minimizing or avoiding student 



According to the CMCD Newark Report for the 1998-99 school year, 19 elementary and six high school 
teachers were selected to serve as CMCD school-based advisors. In the fall of 1999, the CMCD program office also 
hired a part-time lead coordinator to help manage implementation support activities. The entire group of Newark- 
based CMCD staff and school-based advisors convene monthly at the CMCD program office, which is housed in one 
of the Project GRAD Newark schools. 

^’Classroom observation data, which were provided by CMCD program staff, are based on 239 classroom ob- 
servations conducted in seven Project GRAD Newark schools in the spring of 1999. These data are quantitative (that 
is, are based on a checklist of CMCD materials and techniques observed in the classroom) and do not capture the 
quality of program implementation. However, confidential narrative feedback is provided to each teacher several 
times a year. 
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disruptions through early planning of and dialogue about expected classroom behavior), caring 
(creating a respectful classroom environment that is equitable for all), cooperation (building stu- 
dents’ perceptions that they are trusted and responsible members of the school community), or- 
ganization (establishing routine procedures and roles for students to enhance efficiency and order 
in the classroom), and community (fostering ongoing contact between teachers and parents).^® 
These themes are reflected through practice that is visual in nature, fueled largely by the use of 
CMCD materials or “artifacts” that can be seen, touched, or otherwise experienced through ac- 
tion by teachers and students who are implementing the program. Figure 3.2 shows the percent- 
age of classrooms in Project GRAD Newark schools that were implementing CMCD techniques 
in each theme area.^® It should be noted that CMCD artifacts were frequently posted outside the 
classroom, as were motivational messages, reminders, and photos related to the program — all of 
which were generally also displayed in Project GRAD Newark schools on bulletin boards (near 
schools’ main entrances) and in hallways.**® Such postings acted as prominent signals of CMCD 
implementation in the building, as evidenced by the following comment by a student: “Consis- 
tency Management is important. ... You see it all around the school.” 

Formal implementation visits are conducted by CMCD consultants from the University of 
Houston and CMCD staff developers. Administrative and teaching staff are given feedback (in 
written and graphic form) on the strengths and next steps of CMCD implementation in their 
school, as well as information on CMCD in the whole feeder system. The designed evolution of 
CMCD implementation is intended to solidify classroom-level program application in the first 
year. In the second year, efforts then extend to implementing CMCD in critical common areas 
outside the classroom, such as in hallways, the cafeteria, and playgrounds.**' To achieve this goal, 
CMCD training is provided to paraprofessional staff (that is, aides, office staff, security guards, 
and food service workers) in each school. In the Project GRAD Newark schools, CMCD training 
for paraprofessional staff began in September 1998. However, CMCD workshop participation 
among these staff members has been much lower than that among teachers and school adminis- 
trators. Only 57 percent of noncertified staff participated in program training in the fall of 1998; 
21 percent attended follow-up training in the fall of 1999.**^ In the 1999-2000 school year, 
CMCD program staff plan to provide targeted assistance to strengthen program implementation 
in common areas of the school. 

• CMCD staff have engaged school administrators to serve as leaders of the 
program’s implementation process. 



^®For a more detailed definition of these themes, see Freiberg, 1996. 

^®CMCD techniques that fall under the community theme were not assessed at the Project GRAD Newark 
schools in the 1998-99 school year. These activities were assessed in the 1999-2000 school year, however, and 
CMCD staff are expected to report on these implementation levels in 2000. 

^CMCD staff developers routinely update these postings. 

^'CMCD staff will begin to assess program implementation beyond the classroom formally in the second year of 
program implementation. This finding is based on informal MDRC observations in the Project GRAD Newark 
schools. 

‘*^Lead CMCD program staff report that because NPS does not allocate staff development days for paraprofes- 
sional staff, workshop attendance is optional, which has resulted in lower participation rates. CMCD staff report that 
they and Newark district officials are currently discussing the possibility of scheduling staff development days for 
paraprofessionals during the school year. 
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Figure 3.2 

Implementation of CM CD Themes in Project GRAD Newark Schools, 

Spring 1999 




SOURCE: These data were obtained from Dr. H. Jerome Freiberg. 

NOTES: These data are based on 239 classroom observations conducted in seven Project GRAD schools (all 
but Dayton Street School and Malcolm X Shabazz High School, which began CMCD implementation later) in 
the spring of 1999. 

The CMCD community theme was not assessed at the Project GRAD Newark schools during the 1998-99 
school year. 

Some examples of CMCD strategies and artifacts include the Classroom Manager, a rotating position that 
gives students a greater role in the operations of the classroom by allowing them to be responsible for 
completing tasks such as erasing the board and managing absence packets; the Exit Ticket, which involves 
students' writing down what they have learned at the end of the day while listening to classical music so that the 
students can end the day on a calm note; and the Go-Around Cup, which ensures greater variability and equity 
in questioning students in the classroom. 
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Principals and vice principals across the Project GRAD Newark schools have convened 
with CMCD lead staff regularly since March 1998 to brainstorm, share implementation ideas and 
accomplishments, and discuss concerns about and next steps in program development.'*^ To pro- 
mote data-driven decision-making among principals, at each workshop CMCD staff present re- 
search data — which they compile for individual schools — on rates of staff participation in 
CMCD training, levels of program implementation, and number of student discipline referrals. 
SLT in administrators and Project GRAD Newark, Inc., staff often attend these CMCD principal 
leadership workshops. A total of seven principal leadership workshops have been held thus far. 
Principals are expected to attend all CMCD teacher training workshops and to support 
implementation at their schools by modeling the program’s techniques. For example, one 
principal commented: “When I walk into a classroom and start using the Consistency 
Management model, then the teacher will realize, ‘Oh, that’s what I should be doing.’ What I am 
really saying to them is come over a little closer to CMCD.’’ 

Another example of Project GRAD Newark principals’ taking the lead in promoting 
CMCD implementation is their collective effort to establish a new discipline referral form (and 
process) across the Project GRAD Newark schools.'*'* The new form was designed jointly by the 
Project GRAD principals (in collaboration with CMCD lead staff) and introduced for implemen- 
tation in the fall of 1998. The form serves as a coaching tool that encourages teachers to resolve 
minor disciplinary infractions (such as tardiness, horseplay, and refusal to participate) within the 
classroom using CMCD techniques.'*^ Only students who have repeatedly violated established 
codes of conduct or committed a major infraction (such as fighting) are referred to the principal’s 
office. CMCD program staff recommended that students with severe social or home-related prob- 
lems be referred to the school’s CIS program office. When used properly, the new discipline re- 
ferral process is intended to lower the frequency with which students are removed from the class- 
room for disciplinary reasons, thus reducing the amount of time they spend waiting in the princi- 
pal’s office instead of learning.'*^ 

During the year in which this new discipline referral process was put in place, there was 
confusion among staff about how to use the form. Although the following was certainly not the 
intended effect, some teachers expressed concern that they would be reprimanded if they made an 
office referral and, therefore, sometimes decided to handle student disruptions that warranted 



“’^When Malcolm X Shabazz High School adopted CMCD, department chairs were invited to attend these meet- 

in no Qo 

'^^MDRC telephone interviews with administrators at seven of the Project GRAD Newark schools in April 1998 
revealed that, prior to CMCD, procedures for student discipline referrals varied considerably within and across 
schools. A few schools had carefully spelled out referral procedures — which required forms for all discipline refer- 
rals — that teachers and administrators followed. Most schools, however, had informal procedures, such as verbal 
referrals or notes written on whatever paper the teacher had at hand. To varying degrees across schools, records were 
kept on discipline referrals made during the year; only one school maintained files on discipline referrals from previ- 
ous years. 

'‘^According to CMCD program staff, Newark principals requested development of a new discipline referral 
form. This activity is not ordinarily part of the CMCD program. 

'^^Copies of discipline referral forms are collected and tallied monthly by CMCD program staff, who provide 
principals with an analysis of their school’s data. An analysis of CMCD discipline referral data is presented in 
Chapter 4. 
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more than a discipline referral form in other ways.'*^ Another teacher concern was the perceived 
lack of follow-up services or administrator action in cases where discipline form documentation 
showed that teachers’ repeated attempts at dealing with a particular student’s disruptive behavior 
had not worked. School administrators, too, did not feel that the referral process was being used 
consistently within their schools. CMCD program staff have responded quickly to these concerns 
by helping principals revise the discipline form and reviewing its procedures thoroughly at the 
CMCD teacher workshops in the fall of 1999. Although staff are still adjusting to using the new 
form, filling it out and following record-keeping procedures developed with the assistance of 
CMCD staff are becoming the standard steps that teachers take to handle student discipline in 
Project GRAD Newark schools. 

Communities in Schools (CIS) 

CIS project directors have been operating in the Project GRAD Newark schools since 
September 1998.'** The responsiveness of CIS staff to individual schools’ needs for services pro- 
gramming helped gamer immediate support for the program from Project GRAD Newark 
schools. Through the efforts of CIS staff, the CIS program broadened schools’ access to an array 
of student, parent, and family services, as well as to enrichment activities such as educational 
field trips and clubs. CIS project directors have also taken the lead in coordinating family support 
activities for SFA and special Project GRAD Newark events. 

• CIS operations were successful in accessing services and special programs 
tailored to specific needs in each Project GRAD Newark school. 

CIS project directors used an approach developed by the national CIS program office as a 
student services model to identify and secure resources to design a program that meets the needs 
of their individual schools.'*^ In some schools, the priority was parent involvement activities. 
Program priorities in other schools ranged from student counseling services to after-school pro- 
grams to academic enrichment field trips. The customized nature of CIS appealed to school staff 
who felt that the program provided a way for them to connect students (especially those with 
problems) to additional sources of support. Several principals viewed the flexibility of CIS re- 
sources as the most beneficial aspect of CIS overall. 

Table 3.1 presents information about the types and number of CIS activities conducted 
during the 1 998-99 school year, as well as the number of people who participated in each. The 



‘‘’Alternatives to office discipline referrals included making referrals to CIS staff, sending disruptive students to 
another teacher’s class, or “teaching over it.” There was no evidence that punitive actions were taken against teachers 
for making office referrals. 

^®Prior to this initiative, CIS academy programs were operating in two Project GRAD Newark schools: the Bur- 
ger King Academy at Malcolm X Shabazz High School and the Giants Academy at Louise A. Spencer. Both acad- 
emy programs were targeted at a subset of students. Under Project GRAD Newark, the CIS program is being imple- 
mented school-wide in all project schools. School-wide CIS initiatives operate alongside the preexisting academy 
program at Louise A. Spencer. 

‘‘’Project directors were encouraged to use CIS of New Jersey’s agency partnership resources to gain access to 
local service providers; they also created a network among themselves for sharing program contacts. Each director 
was allocated a budget to use in acquiring programs for their respective schools. 
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Table 3.1 



CIS Activity Report (Cumulative) for Project GRAD Newark Schools, 

September 1998-June 1999 



Activity 


Number of Students 
(or Parents) Who 
Participated 


Number of 
Events 


Cultural enrichment activities 


8,926 


106 


Field trips 


3,199 


59 


Career counseling, individual counseling, and mentoring 


2,130 


- 


Home visits and parent conferences 


680 


- 


Community service 


676 


- 


Tutoring 


553 


- 


Family events 


417 


28 


Referrals to community agencies^ 


48 


- 



SOURCE: These data were obtained from the CIS of New Jersey program office. 

NOTES: Data for Madison Avenue School and Miller Street School were available only for the months of 
May and June. People who participated in multiple activities are counted more than once. 

^This row includes social service organizations, medical and substance abuse facilities, and recreational 
and educational facilities. 
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activities included cultural enrichment, field trips, student counseling and mentoring, tutoring, 
home visits and parent conferences, family events, student community service, and referrals to 
community agencies (such as social services or recreational facilities). Most students in the Pro- 
ject GRAD Newark schools participated at least once in the cultural enrichment activities and 
field trips, of which 165 were conducted. Through CIS programs. Project GRAD Newark stu- 
dents also availed themselves of career counseling, individual counseling, mentoring, and tutor- 
ing services.^® Student referrals to community agencies were also made, but were less frequent 
than other CIS activities. The families of Project GRAD Newark students had contact with CIS 
through family events, home visits, parent awareness and skills development activities (such as 
GED classes and seminars about high school and college requirements), and parent conferences 
to encourage parent involvement. Nearly 90 activities, which were participated in by 1,000 Pro- 
ject GRAD Newark families, were directed toward these services. 

• In addition to coordinating activities tailored to school-identified needs, 

CIS staff have played a major role in managing other Project GRAD ac- 
tivities, as requested by Project GRAD Newark, Inc., staff. 

At the request of Project GRAD Newark, Inc., staff, CIS project directors were given re- 
sponsibility for planning and coordinating major events supporting implementation of the GRAD 
scholarship component, such as Project GRAD Day and the Walk for Success.^’ Also, the CIS 
director at Shabazz High School has recently become GRAD scholarship manager and is work- 
ing to strengthen GRAD scholar recruitment and monitoring activities.^^ Finally, CIS directors at 
Project GRAD Newark elementary and middle schools have been assigned the role of Family 
Support Team leaders,^^ which requires them to make a substantial time commitment. All these 
activities (for example, parent outreach, student engagement, and social service supports) are 
consistent with the CIS program’s key goals. However, as CIS operations continue to absorb 
these activities, there is a risk that the program will strain its capacity to operate effectively. 

Success for All (SFA) 

SFA activities are aimed at restructuring reading curricula and instmction in grades Pre-K- 
6.^“* Given that implementation of SFA in the Project GRAD Newark schools began in Septem- 
ber 1999, SFA could not have had an impact on the reading test scores reported below because 
these data are based on tests administered before that time. Although the staff orientation in SFA 
was well conducted, unanticipated aspects of program implementation revealed a need for addi- 
tional SFA training and curricular materials. Early feedback from Project GRAD Newark schools 
indicates that students are responding well to the SFA structure (that is, that students feel more 
confident about reading when grouped by reading level). School-level implementation has pro- 

“These statistics are cumulative; that is, families receiving multiple services are counted more than once. 

*'The CIS project directors established a Walk for Success planning committee, which met weekly for six 
months prior to the November event. 

*^e CIS director was hired as the scholarship manager in February 2000. A new CIS director and an assistant 
were then named. 

^^Family Support Teams are part of the SFA program component. 

^Students in Pre-K and Kindergarten are immersed in the SFA program throughout the day; students in grades 
1-6 receive 90 minutes of SFA instruction daily. 
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ceeded with intensive, ongoing guidance from SFA trainers as well as from district and state 
technical assistance providers. The school district pays most of the costs of implementing SFA. 

• Project GRAD Newark principals became a driving force in preparing 
their schools to implement SFA. 

Staff at the Project GRAD Newark schools began preparing for SFA program implemen- 
tation during the 1998-99 school year. To varying degrees, school staff participated in program 
preparation activities, which included attending SFA orientation sessions, visiting other SFA 
schools, collecting and distributing articles and other information on SFA from the Internet, and 
attending conference workshops on SFA. To secure the minimum rate of teacher approval re- 
quired to adopt SFA (80 percent), the voting process was staggered across Project GRAD New- 
ark schools to allow teachers the time they needed to render an informed vote. In cases where the 
initial vote did not meet the 80 percent threshold, additional orientations were held to raise 
teacher support for the program. All the Project GRAD Newark elementary and middle schools 
eventually reached the minimum threshold. Having begun in February 1999, the SFA voting 
process ended at all Project GRAD Newark schools by May 1999. 

Project GRAD Newark principals took the lead in creating a supportive structure for SFA 
implementation by modifying the operational and physical environment of their schools. Changes 
instigated by principals were most apparent in the areas of budgeting, daily scheduling, school 
facilities, and classroom assignments. Principals began working with their School Management 
Teams in the spring of 1999 to draw up a budget to cover SFA program expenditures, a process 
that involved consolidating and/or reallocating existing funds as well as soliciting new funds 
through outside grants. In the summer of 1999, principals (along with SFA facilitators and Fam- 
ily Support Team leaders) attended a week-long national conference on SFA. During the sum- 
mer, Project GRAD Newark principals restructured their schools’ daily schedule, having been 
given autonomy in determining the time of day SFA would be implemented. Some principals 
scheduled SFA from 8:30 to 10:00; others scheduled it from 9:30 to 11:00; and one scheduled 
two staggered SFA reading periods for students at different grade levels — from 9:30 to 11:00 
for students in grades 1-4 and from 10:25 to 1 1:55 for students in grades 5-6. In addition, princi- 
pals modified classroom assignments and school facilities to accommodate the program. For ex- 
ample, one principal changed 21 classroom assignments in order to facilitate SFA implementa- 
tion; at another school, staff worked all summer refurbishing a basement area that was to become 
the SFA tutoring room. 

• Preliminary feedback from teachers suggests that they have a positive 
view of the program’s structure and students’ reactions to it, but have 
experienced difficulty in adjusting to the pacing of instruction required 
for SFA implementation.^^ 

Teachers liked the structure of SFA, especially the fact that students are grouped by reading 
level. They felt this type of grouping diminished in-class competition and gave students at all read- 



^^Teacher feedback sessions and classroom observations of SFA implementation were conducted by SFA pro- 
gram staff in the fall of 1999; MDRC researchers observed the feedback sessions at six of the Project GRAD Newark 
schools. These observations form the basis of the teacher reports presented here. 



ing levels an opportunity to feel successful in reading. Teachers also liked the SFA materials. Class- 
room observations in the Project GRAD Newark schools by SFA program staff found evidence of 
SFA implementation in teachers’ posting of SFA storybook characters, students’ writing samples 
and artwork, and wall displays of vocabulary words in SFA stories. Teachers also frequently used 
SFA classroom management hand signals to prompt and praise students during instruction. How- 
ever, teachers struggled with some of the demands of SFA programming — for instance, its rapid 
pace, getting students to work independently in learning centers, keeping classroom displays up to 
date with the SFA lesson currently being taught, and mainstreaming English as a Second Language 
(ESL) students and special education students into SFA reading groups.^^ 

• Results of the initial reading test (administered for the SFA program be- 
fore SFA implementation) indicated that a large percentage of students in 
grades 2-6 were reading at the l^‘-grade level. 

Although 1®‘ graders represented just 15 percent of Project GRAD Newark students tested 
for the SFA program, 51 percent of students tested overall were reading at the l®‘-grade level.^^ 
These results led to a high demand (in all Project GRAD Newark schools except the high school) 
for the SFA Older Roots curriculum, which is designed for students in grades 3-6 reading at a 1®‘- 
grade level. Unfortunately, few teachers had been trained in this curriculum approach. In order to 
meet the need for Older Roots instruction, several schools had to catch up (by conducting teacher 
training and ordering additional curriculum materials) as they were starting SFA implementation. 
Requirements for SFA tutoring activities have also exceeded initial expectations and allocated 
resources.^* Project GRAD Newark, Inc., is assisting the district’s efforts to build schools’ capac- 
ity to meet SFA tutoring requirements by providing funds to hire and train additional “permanent 
substitute” teachers to be SFA tutors (and to teach during the 90-minute reading period) in the 
Project GRAD Newark schools. The addition of these staff will allow more students to receive 
daily tutoring and will also help reduce the size of SFA reading groups. 

• SFA professional development activities have faced time constraints. 

The initial SFA training workshop took place over three days immediately following the 
close of the 1998-99 school year. The timing of the workshop posed some problems. Participants 
reported feeling tired, having released students for the summer just the day before. And the 
length of the initial workshop — three full days — put teachers, who were in the midst of com- 
pleting year-end administrative activities, under considerable time pressure. Finally, the time lag 
between initial training and SFA implementation was two months, and during the summer break 
there was no opportunity for teachers to review SFA curricular materials or apply SFA concepts 

^*The Newark Public School District’s central office has recommended that schools mainstream special educa- 
tion students and ESL students into SFA instruction. The district has not purchased any Spanish-language SFA mate- 
rials, so bilingual students are immersed in English-only SFA reading groups. Students do not participate in SFA 
only in special circumstances — for instance, if they have tested far below even the lowest SFA reading level. 

^’in order to determine the assignment of students to SFA reading groups, a reading assessment is administered 
every eight weeks. For the initial assessment, a total of 3,380 students in grades 1 -6 were tested. 

^*The SFA program recommends that 30 percent of graders, 20 percent of 2"‘‘ graders, and 10 percent of 3"* 
graders receive tutoring services. The Project GRAD Newark schools had anticipated allocating tutoring services 
only for the grade. However, the learning needs revealed by the tests indicated that more tutoring than anticipated 
was required. 
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in the classroom setting.^® A second introductory SFA training workshop was conducted in Au- 
gust, but was targeted at teachers who were hired over the summer and newly assigned to Project 
GRAD Newark schools.^® 

Once these difficulties were discovered, it was decided that half-day SFA refresher work- 
shops would be conducted at each school before the start of the 1999-2000 school year. How- 
ever, according to two SFA trainers, these sessions were not intended to cover curricular materi- 
als as thoroughly as school staff wished.^’ During the fall semester, additional training work- 
shops for new teachers were incorporated into the implementation site visits conducted by SFA 
trainers. A second round of SFA training for all teachers took place during district staff develop- 
ment days in January 2000. 

Full-time SFA facilitators are in place at each school implementing the program. In al- 
most every case, the SFA facilitator is the former literacy staff developer for the school. The fa- 
cilitators are teachers and, as such, assist other teachers in implementing the SFA program in a 
collegial (rather than supervisory) capacity. In the words of one SFA program staff member, the 
facilitators “have the toughest role” and the greatest adjustment to make in implementing SFA 
because they have to work differently than they did as literacy staff developers. One school 
administrator described the difference this way: 

The literacy staff developer made sure that staff as well as children were pre- 
pared for ESPA and GEPA.^^ Their job was focused on test preparation. The 
facilitator’s role is to make sure the SFA program is running smoothly — that 
teachers have SFA materials, the SFA assessment is administered and ana- 
lyzed properly, [and] they make sure SFA tutors have their program in place. 

Their focus is not so much on the state test. . . . The shift in focus leaves them 
somewhat vulnerable because they know there are exact things they have to do 
that will be judged by the SFA monitoring process. [The facilitators] feel it’s 
different and they’re overwhelmed. 

These school-based facilitators are the lead professional development providers for the 
SFA program. They are expected to conduct classroom observations, meet frequently with teach- 
ers (in groups and individually) to reflect on teaching and learning practices of the SFA program, 
review reading assessment results with teachers every eight weeks to target teaching strategies, 
and work closely with principals to maintain support structures and resources for effective im- 
plementation. However, early experiences in the Project GRAD Newark schools indicate that the 



^’Because teachers were not given the SFA teacher’s manual at the June workshop, they could not readily review 
the program’s curriculum during the summer on their own. 

*^e Newark public schools held an SFA introductory training workshop for new teachers who had been hired 
as of August 1999 (and returning teachers who missed the June workshop). The number of new teachers may have 
been unusually high this year because, in September, Newark received a state-funded grant aimed at class-size reduc- 
tion in the primary grades through the hiring of more teachers. 

*'These SFA trainers pointed out that the intensive three-day initial training typically takes place at the start of 
the school year, immediately prior to implementation. 

*^The Elementary School Proficiency Assessment (ESPA) is New Jersey’s state-mandated achievement test ad- 
ministered in grade 4; the Grade Eight Proficiency Assessment (GEPA) is New Jersey’s state-mandated test adminis- 
tered in grade 8. ESPA and GEPA results are used to assess both student and school performance. 
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time available for these professional development activities (especially teacher-to-teacher meet- 
ings) is extremely difficult to find within the daily school schedule. Funds to give teachers re- 
lease time for planning during the day have been strained, making hours before and after school a 
frequently used last resort for scheduling the necessary professional development time.®^ 

• The expertise and accessibility of SFA program trainers have been in- 
strumental in the development of school-level SFA implementation. 

SFA trainers have responded to staff in the Project GRAD Newark schools by answering 
questions, observing SFA reading groups and tutoring activities, providing suggestions and mate- 
rials, and otherwise guiding ongoing implementation. Two SFA lead trainers have been assigned 
to work with school staff — especially with principals and SFA facilitators, with whom they 
communicate weekly, often daily. One of the lead trainers periodically convenes a study group of 
SFA facilitators who work in Project GRAD Newark, which met three times in the fall of 1999. 
The lead trainers also give staff their home phone numbers and encourage them to call with ques- 
tions or problems at any hour; as one school administrator remarked, “They are at the [SFA] fa- 
cilitators’ beck and call.” Accompanied by a team of SFA program staff, the SFA lead trainers 
visit the Project GRAD Newark schools several times per year to conduct intensive, on-site re- 
views of SFA implementation.^ During each two-day visit, the team observes SFA reading and 
tutoring instruction and conducts small-group feedback sessions with staff.®^ 

There is a local program management infrastructure in place to support SFA implementa- 
tion as well.^ For example, the SLT HI Literacy Supervisor attends SFA teacher training (as do 
other district office staff) and spends much time securing SFA materials for schools and meeting 
one-on-one with SFA facilitators. The SLT HI office also sponsored an “SFA Quality Leadership 
Workshop” for principals and SFA facilitators, which further reviewed program management 
strategies. In addition. Project GRAD school staff have access to the New Jersey Success for All 
Network (sponsored by the New Jersey Department of Education), the purpose of which is to 
bring SFA school staff together so that they can share strategies and address common concerns 
regarding SFA implementation. 

The strong local interest in and support for SFA implementation from the outset have been 
advantageous for Project GRAD Newark schools. On the other hand, this overlap between the Pro- 
ject GRAD Newark initiative and the whole-school reform mandated by the state — which requires 
schools to select a reform model — has caused some confusion among school staff. For example. 



*^The Newark Teachers Union (NTU) agreement with the district stipulates that teachers be paid for all man- 
dated staff development time. To varying degrees at each school, teacher participation in unmandated SFA profes- 
sional development activities has been affected by this agreement. 

®^In January 2000, the Success for All Foundation assigned an additional staff member to help the two SFA lead 
trainers to coordinate implementation visits and schedule training sessions for the Project GRAD Newark schools. 

**Because of needs identified by the Project GRAD Newark schools, the site visit team also incorporated train- 
ing for new teachers into their schedule of activities. A few weeks after each visit, the team submits a written report 
covering implementation assessment and issues to the school and the district. Thus far, two rounds of implementation 
visits have been completed in most Project GRAD Newark schools: two additional rounds are scheduled to take 
place before the end of the school year. 

**This is in large part because the SFA program component of Project GRAD Newark is also one of the state- 
mandated whole-school reform choices for the Abbott districts. 
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some principals and teachers were initially uncertain whether Project GRAD Newark schools had 
to select SFA as their whole-school reform model and about what proportion of funding for SFA 
would come from Project GRAD Newark, Inc., the state, and local school budgets.®^ 

Challenges Ahead 

So far, this chapter has highlighted key accomplishments, processes, and issues that have 
been important to the development of the Project GRAD Newark initiative. Project GRAD New- 
ark, Inc., has rallied key stakeholders to support this education reform; the Newark school district 
has made Project GRAD program components integral parts of the Newark school reform 
agenda; and school-level implementation has proceeded on schedule with projected timelines. In 
view of the initiative’s implementation record during its brief tenure to date (24 months), there is 
cause for optimism. However, because the Project GRAD Newark initiative is still in the early 
stages of implementation (it takes four years to implement the program fully in a feeder system), 
it has progressed only part of the way to achieving its expressed goals. 

As stakeholders in the Project GRAD Newark initiative continue to build and strengthen 
its implementation, they might bear in mind the following suggestions for improving school 
functioning and student outcomes. The MDRC evaluation of Project GRAD Newark’s early im- 
plementation points to two major challenges that deserve immediate attention. The first is the 
need to strengthen the quality of curricula and instruction in grades 7-12. The second is the need 
to develop communications and professional development efforts that focus on the operational 
linkages among the program components that make up Project GRAD Newark. 

• There is a need for Project GRAD Newark, Inc., and the district to work 
together to strengthen the curricular focus of the initiative in grades 7-12. 

As discussed earlier, the Project GRAD model is based on a feeder pattern approach that 
takes into consideration the entire K-12 grade span. CMCD and CIS operate in all schools and at 
all grade levels in the Project GRAD feeder pattern, from Pre-K through grade 12, providing 
classroom management techniques and social services that enable teachers to run more self- 
disciplined classrooms and that afford students access to a variety of supports. However, the cur- 
ricular components of Project GRAD — that is, the components that focus specifically on teach- 
ing and learning (that is, the Success for All Foundation’s reading and math curricula programs) 
— target only Pre-K through grade 6.®* This feature leaves a span of six grades (7-12) in which 
Project GRAD Newark depends largely on schools’ existing curricular programs in the core aca- 
demic areas of reading and math to prepare students for college enrollment. Without a strong cur- 



®’lnitially, school administrators anticipated that Project GRAD Newark, Inc., would directly or indirectly pay 
most SFA expenses (such as those for professional development and materials). However, because SFA was selected 
as the schools’ choice under the state-mandated whole-school reform regulations, its implementation is funded 
largely through the district and school-level budgets. Project GRAD Newark, Inc., has provided supplemental fund- 
ing to support SFA in areas that have been identified by SLT III and staff at the Project GRAD Newark schools. For 
example. Project GRAD Newark, Inc., provided funds for the district to purchase the reading test (Gates McGinitie) 
that is used to determine students’ placement in SFA reading groups. 

®*The SFA reading program is implemented in grades Pre-K-6. Project GRAD Newark schools are expected to 
begin implementing the MathWings curriculum in grades 1-5 during the 2001-02 school year. 
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ricular focus in grades 7-12, the initiative risks erosion of the academic advantage that students 
may achieve as a result of Project GRAD in the early grades. 

Malcolm X Shabazz High School has adopted the Talent Development High School 
Model, which is curriculum driven, in addition to the Project GRAD Newark model. The school 
began the first stage of this reform model — the 9*-grade Success Academy — at the start of the 
1999-2000 school year. Although this action provides a foundation on which to build, a key task 
for Project GRAD Newark, Inc. (in conjunction with district staff) is to examine student 
achievement issues and schools’ existing and anticipated curricular offerings (and other academic 
enhancements) for grades 7-12. This information can serve as a basis for identifying implications 
and possible next steps for the initiative. 

• There is a need to improve coordination across the various Project GRAD 
Newark program components; this will maximize each program compo- 
nent’s contribution to achieving the initiative’s goals. 

As discussed earlier, three of the five Project GRAD program components (CMCD, CIS, 
and SFA) existed as stand-alone programs in other schools prior to their integration into the Pro- 
ject GRAD model.®^ Each of these programs has organizational goals and an identity separate 
from Project GRAD and is represented by a lead agency or developer that provides schools with 
materials, professional development, technical assistance, and other resources designed to sup- 
port implementation. Therefore, representative staff and trainers from these stand-alone programs 
are accustomed to operating independently of each other in different cities and contexts. For 
example, the CMCD program, the lead staff of which is based at the University of Houston, has 
been adopted by 107 schools nationwide and abroad (in Italy and the Netherlands), and many of 
these schools are Project GRAD sites.’° CIS is a national organization with programs operating 
in 1,500 schools in over 32 states. The Success for All Foundation is also a national organization, 
and the SFA reading program is now operating at over 1,500 schools in 47 states, as well as 
abroad. 



The implementation timeline for Project GRAD Newark reflects a staggered approach to 
introducing these stand-alone programs into schools, but as of September 1999, four program 
components were operating concurrently within the Project GRAD schools.’’ With this degree of 
new program influx and overlap at the school level, issues related to the operational linkages be- 
tween, or program coherence of. Project GRAD Newark’s components are surfacing. Prominent 
education researcher Fred Newmann argues that program coherence is key to establishing “[the] 
clarity, focus, and continuity” needed to effectively sustain school resources and staff commit- 
ment for the larger goals of reform, but acknowledges that it is the “most troublesome” for im- 
plementors to achieve. 



^’The only component that originated from the Project GRAD model is the GRAD scholarship guarantee. As a 
result, its activities are not developed or represented by any technical assistance provider apart from Project GRAD. 

’^Independent of Project GRAD, CMCD is slated to expand to other schools in the Newark district. 

”As mentioned earlier in this report, the Project GRAD Newark implementation timeline began with the an- 
nouncement of the scholarship guarantee and the start of the CMCD program in early to mid 1998. CIS was added in 
September 1998, and SFA implementation commenced in September 1999. Implementation of a fifth component, 
MathWings, is anticipated to begin in the fall of 2001. 
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At times, it has been difficult for staff of the individual components of Project GRAD New- 
ark to communicate the relationship between their programs’ role and objectives and those of the 
other components. (This was also true during the first two years of Project GRAD implementation 
in Houston.) Nonetheless, some noteworthy implementation integration among the components has 
occurred. CIS project directors, for example, have operated their program in an integrated fashion 
by assuming a leadership role in several activities that support implementation of the GRAD schol- 
arship and SFA components; they also attend CMCD and SFA training workshops. Similarly, 
CMCD program staff have supported the implementation of other Project GRAD components by 
assisting CIS project directors in planning the Walk for Success and Project GRAD Day. In addi- 
tion, at their teacher workshop in October 1999, CMCD staff presented a series of diagrams show- 
ing the overlap between CMCD and SFA concepts and provided a sample reading lesson to model 
strategies that teachers could use to integrate CMCD techniques and artifacts within the SFA pro- 
grarn.’^ However, there is currently no consensus among lead staff of the Project GRAD program 
components regarding the direction or content of ongoing activities (including professional devel- 
opment) aimed at integrating implementation of the various components. 

The staff of Project GRAD Newark, Inc., which has served as the coordinating body for 
the overall initiative, recently convened program component leaders to address the issue of pro- 
gram coherence. This action represents a key step toward strengthening the initiative, especially 
in helping it sort out the complexity of its organizational relationships and address the persistent 
issue of arranging adequate professional development time for each component. There is a con- 
tinuing need for such formal joint planning to coordinate communication, planning, and informa- 
tion-sharing across the various components. 



issue of program coherence is of particular importance for the components of the Project GRAD initiative 
that are largely classroom-based (CMCD and SFA), the implementation of which requires teachers to use specific 
techniques and materials in their classrooms. SFA and CMCD each have distinctive materials and protocols for 
teacher-student classroom interaction, which are necessary for effective implementation of these programs. 
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Chapter 4 

Monitoring Change in School Functioning and Student Outcomes 



Chapter 3 presented findings on the implementation of Project GRAD Newark that ad- 
dress the first major question introduced in Chapter 1. This chapter turns to the second question: 
Are student outcomes improving in the Project GRAD Newark schools even before implementa- 
tion of the initiative’s curricular components? It also briefly touches on the third question: Did 
Project GRAD contribute to this improvement? 

Figure 1.1 (in Chapter 1) outlines the measures of school functioning and student out- 
comes to be tracked over the course of the evaluation of Project GRAD Newark. As part of the 
evaluation, the MDRC research team is constructing a database of historical data on these meas- 
ures for Project GRAD Newark schools, selected comparison schools, and schools in the rest of 
the district (that is, all schools in the district except those implementing Project GRAD). For the 
Project GRAD Newark elementary and middle schools, the comparison schools are the nine 
schools and school annexes in School Leadership Team IH (SLT IH, a geographical cluster of 
schools in the Newark school district that includes all nine Project GRAD Newark schools) that 
are not part of Project GRAD Newark. For Malcolm X Shabazz High School, another compre- 
hensive high school in Newark (Weequahic High School) serves as a comparison school. 

The comparison schools were chosen on the basis of their similarity to the Project GRAD 
Newark schools and the recommendation of Newark public school staff familiar with the dis- 
trict’s schools, with the schools’ instructional emphasis, and with their plans for future reform. 
The Appendix to this report reports on the characteristics of the comparison schools. Some test 
scores for New Jersey districts that are socioeconomically similar to Newark (that is, those in 
District Factor Group A) and for the state as a whole are also presented. As discussed later in this 
chapter, student outcomes in these comparison schools and in schools in the rest of the district 
will be used to track the trends in outcomes that would be expected in the absence of Project 
GRAD’s special services. Some of these data pertain to individual students (such as test scores), 
while other measures pertain to individual schools (such as average attendance rate). By focusing 
on a few specific measures of school functioning and student outcomes, this chapter illustrates 
the type of analyses that this database makes possible. 

Key Findings in This Chapter 

• Because implementation of Project GRAD Newark’s curricular components 
had not occurred before the student achievement data presented here were col- 
lected, the program’s effects on most measures have not yet manifested them- 
selves. 

• The data already available permit the tracking of many key measures, and data 
for tracking other measures can be readily obtained. 
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• With the implementation of Consistency Management & Cooperative Disci- 
pline, school functioning has clearly improved according to one measure, 
namely, the number of discipline referrals to the principal’s office. 

• In the first year of Project GRAD Newark implementation, test scores in 3"*- 
grade math (and to some extent in 3'^^-grade reading) were better than would 
be expected based on historical data. 

• Test scores in other grades have not as yet shown improvement. It should be 
emphasized, however, that the present analysis of test scores covers a period 
predating implementation of all Project GRAD curricular components in the 
Project GRAD Newark schools. The scores analyzed here came from tests 
administered in the fall of 1998 and spring of 1999; Success For All imple- 
mentation began in the fall of 1999. 

For some measures, the present analysis of trends includes data (for example, attendance 
rate, number of suspensions, certain test scores) from a period before Project GRAD Newark im- 
plementation began, while for other measures such historical data are not available. Where data 
on the measures are available for other schools in the Newark school district, the corresponding 
trends in all other schools in the Newark school district and in a set of comparison schools are 
presented. 

Future analysis of outcomes in the Project GRAD Newark schools will be more intensive. 
This discussion should therefore be seen as providing only a preliminary look at trends. In future 
reports on the MDRC evaluation, the analysis of Project GRAD Newark’s impacts on student 
outcomes will, where possible, control for key measures of student characteristics in the Project 
GRAD Newark and comparison schools over time, include measures of student achievement in 
an earlier period to assess the achievement gains produced by Project GRAD Newark (its “value 
added”), and include statistical analyses that more accurately calculate measures of the statistical 
significance of the findings to take into account the fact that students in the study were sampled 
from specific schools and specific school years.* 



Linking the MDRC Evaluation's Focus to Program Realities 

The questions addressed at each stage of the MDRC evaluation will be dictated by the re- 
alities of program operations. At this early stage of Project GR/J5 Newark implementation, the 
evaluation has two main objectives: (1) building a framework for monitoring change in school 
functioning and student outcomes and (2) focusing on aspects of students’ experiences where 
early change is most likely already to have occurred. 



What types of change are most likely to manifest themselves at this early stage of pro- 
gram implementation? Students in the elementary and middle grades were most likely to be af- 
fected by Project GRAD in the 1998-99 school year, the period covered by this trend analysis. 



'in particular, where the data permit, a combination of interrupted time series, hierarchical linear modeling 
(HLM), and analysis of value added will be used to analyze student achievement. For a discussion of this approach, 
see Kemple and Snipes, 2000. 
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During the 1998-99 school year, two components of the reform — Consistency Management & 
Cooperative Discipline (CMCD) and Communities in Schools (CIS) — were in place at all grade 
levels in the Project GRAD Newark middle and elementary schools. Over the course of the year, 
students in these schools heard more about Project GRAD Newark and were urged to make plans 
to become Project GRAD scholars.^ 

So far, Project GRAD Newark is likely to have had only a modest overall influence on 
students in the high school, although its effects on individual GRAD scholars might already be 
substantial. Because the two aspects of Project GRAD that were implemented earliest at the high 
school — the GRAD college scholarship guarantee and the Summer Institute Program — were 
offered only to students in grades 9 and 10 during the period covered by this analysis, students in 
grades 1 1 and 12 in the 1998-99 school year were ineligible for the offer and therefore probably 
largely unaffected by it. Further, CMCD was not implemented at the high school until the spring 
of 1999. Finally, the Talent Development High School reform, which promises to make major 
changes in educational offerings at Shabazz, did not begin until the fall of 1999. 

As a consequence, measures of the overall functioning of the high school (for example, 
attendance rate averaged across students, total number of disciplinary incidents) and measures 
specific to grades 1 1 and 12 (for example, average scores on the state’s grade 1 1 test, graduation 
rates) in the 1998-99 school year should be seen as baseline information against which to track 
future progress. However, some measures specific to grades 9 and 10, such as 9“*- and 10“’-grade 
test scores, could have been affected by the GRAD scholarship offer and Summer Institute Pro- 
gram during this period. 

Measures of School Functioning 

The Project GRAD theory of change — see Figure 1 . 1 and Box 1 .2 in Chapter 1 — sug- 
gests that, together, the various components of Project GRAD Newark will result in improved 
school climate and classroom instruction. As Figure 1.1 shows, the key measures of school cli- 
mate and classroom instruction are: student attendance rate, number of discipline referrals, num- 
ber of suspensions, student attitudes toward school, time devoted to classroom instruction and 
learning (as opposed to management and discipline), and high school course-taking patterns. This 
first evaluation report focuses on the measures for which data predating implementation of Pro- 
ject GRAD Newark are available from Newark school district records — namely, attendance rate 
and number of suspensions — to enable examination of early trends in these measures and fo- 
cuses on discipline referrals because of their expected link to the implementation of CMCD.^ 



^With the implementation of Success for All in grades Pre-K-6 in the fall of 1999, Project GRAD Newark may 
have a greater influence on test scores in the spring of 2000 than on test scores at earlier stages of program imple- 
mentation. 

^School-level data on average student attendance and suspensions are available for several years prior to imple- 
mentation of Project GRAD Newark. (Historical information on discipline referrals, however, is not generally avail- 
able.) Surveys such as that conducted by MDRC in January 1999, which collected data on school climate and func- 
tioning and time devoted to instruction, will be used in the future as well. This will allow an analysis of change in 
these measures during the period of Project GRAD Newark implementation. Future MDRC reports will also present 
an analysis of course-taking patterns and school dropout rates, which primarily pertain to the high school. 
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These analyses are supplemented by analyses of a teacher survey administered by MDRC in the 
Project GRAD Newark elementary and middle schools in January 1999. 

Student Attendance 

One basic measure of the school learning environment is the student attendance rate: If 
students do not go to school, they cannot learn what is taught there. Moreover, a high rate of ab- 
senteeism disrupts the teaching process because students miss instruction in topics on which sub- 
sequent work is based. Student absenteeism is an important issue in the Project GRAD Newark 
schools and the district as a whole. Survey responses from elementary and middle school teach- 
ers at Project GRAD Newark schools reveal that 54 percent felt that student absenteeism was a 
moderate or serious problem in their school. While there was some variation in teachers’ judg- 
ments of the importance of this issue, at least one-third of teachers in six of the seven Project 
GRAD schools surveyed and of teachers at almost all grade levels identified this as a moderate or 
serious problem. 

Table 4.1 displays student attendance rates for Project GRAD Newark schools, the com- 
parison schools, and the Newark school district as a whole over a five-year period. In this analy- 
sis, the attendance rate is defined as the percentage of enrolled students who are present on an 
average school day. 

• Attendance rates in the Project GRAD Newark elementary and middle 
schools tended to be slightly above 90 percent and showed little change in 
1998-99, the school year in which program implementation started. 

The attendance rates at Project GRAD Newark elementary and middle schools were gen- 
erally stable over the four years prior to and the one year following program implementation and 
were comparable to the district average for elementary and middle schools. In the 1998-99 school 
year, all the Project GRAD Newark schools had attendance rates above the 90 percent standard 
set by the state.'* Attendance rates in the elementary and middle comparison schools and in the 
district as a whole showed a similar pattern of stability over the five years shown in the table. 

• Attendance rates at Malcolm X Shabazz High School have shown some 
instability from year to year, with a decline in attendance in the 1998-99 
school year. 

The average attendance rate at Malcolm X Shabazz High School over this five-year pe- 
riod falls below the state target rate of 90 percent and below the district and state high school av- 
erages, suggesting that attendance may be a special problem at this school. Further, over the same 
period attendance rates in the comparison high school and in schools in the whole district have 
shown an upward trend. As mentioned earlier, the decline in attendance at Shabazz in 1998-99 
should be seen as part of the baseline against which future trends can be judged rather than as an 
effect of Project GRAD Newark. But these findings highlight the importance of increasing atten- 
dance rates to help improve the educational performance of students in this school. 



‘*This state standard is one of the conditions for returning the Newark public schools to local control. See New- 
ark Public Schools, 1999. 



Table 4.1 



Attendance Rate at Project GRAD Newark Schools, Comparison Schools, 
and Schools in the Rest of the District, by School Year 





School Year 


Change Between 
1997-98 and 1998-99 


School 


1994-95 1995-96 1996-97 1997-98 1998-99 


(percentage point 
difference) 



Project GRAD Newark Schools 
Elementary 



Avon Avenue 


90.1 


89.4 


91.2 


91.8 


91.4 


-0.4 


Belmont Runyon 


92.0 


91.3 


92.3 


94.5 


93.4 


-1.1 


Dayton Street 


91.6 


90.7 


92.3 


90.5 


90.1 


-0.4 


Louise A. Spencer 


91.1 


90.4 


89.8 


90.7 


91.2 


0.5 


Madison Avenue 


92.5 


92.7 


93.2 


93.6 


93.7 


0.1 


Miller Street 


92.6 


92.5 


93.5 


94.2 


93.0 


-1.2 


Peshine Avenue 


91.6 


91.1 


93.4 


93.4 


92.9 


-0.5 


William H. Brown 


90.3 


89.9 


90.7 


92.1 


91.6 


-0.5 


Secondary 


Malcolm X Shabazz 


79.1 


79.2 


77.1 


80.7 


77.4 


-3.3 


Comparison Schools 
Elementary 


Bragaw Avenue 


91.9 


90.8 


93.1 


92.9 


92.6 


-0.3 


Bruce Street 


87.8 


90.2 


91.6 


92.3 


92.5 


0.2 


Chancellor Avenue 


93.0 


91.5 


93.8 


93.7 


92.7 


-1.0 


Chancellor Annex 


92.1 


92.5 


93.3 


93.3 


92.8 


-0.5 


Clinton Avenue 


91.6 


92.5 


91.5 


93.3 


93.1 


-0.2 


G.W. Carver 


91.6 


92.1 


93.2 


93.3 


92.5 


-0.8 


Hawthorne Avenue 


89.9 


89.5 


91.7 


92.6 


92.4 


-0.2 


Maple Avenue 


92.3 


92.1 


92.6 


94.0 


93.6 


-0.4 


Maple Annex 


92.9 


92.9 


95.1 


94.4 


94.1 


-0.3 


Secondary 


Weequahic 


74.8 


74.4 


78.4 


79.6 


82.4 


2.8 


All Newark Elementary Schools 


92.0 


92.0 


92.0 


93.1 


92.5 


-0.6 


All Newark Secondary Schools 


80.0 


80.0 


82.0 


83.6 


83.7 


0.1 



SOURCE: These data were obtained from the Newark Public Schools Office of Student Information 
Services. 

NOTE: The attendance rate is defined as the percentage of enrolled students, on average, who are present 
each school day. 
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Student Discipline Referrals and Suspensions 

The MDRC survey of teachers at the Project GRAD Newark elementary and middle 
schools in January 1999 revealed that about three-fourths of all respondents felt that the level of 
student misbehavior interfered with their teaching and about one-third agreed that rules for stu- 
dent behavior were not consistently enforced in their school. Project GRAD includes components 
(CMCD and CIS) designed to help address this problem by offering teachers new approaches to 
encouraging and supporting improved student behavior — specifically, by equipping school staff 
with new tools for responding to discipline problems and by providing students with social ser- 
vices to address some of the sources of discipline problems. Thus, there is reason to think that 
Project GRAD might lead to improvements in student behavior. 

The MDRC evaluation uses the number of discipline referrals to the principal’s office and 
the number of suspensions as quantitative measures of the frequency of serious student miscon- 
duct and of student removal from the classroom because of misbehavior. Both measures are sub- 
ject to school- and district-level administrators’ policy choices concerning how discipline refer- 
rals and suspensions should be used in daily practice. Thus, a change from one year to the next 
may reflect either a difference in the frequency of discipline problems or the introduction of a 
new policy for responding to such problems. However, if similar changes are observed on multi- 
ple measures, then they are less likely to be the result of policy changes. Thus, examining trends 
in discipline referrals and suspensions provides both useful information about the context in 
which Project GRAD Newark is being implemented and a means of measuring the program’s 
effect on discipline problems, the amount of teacher and administrator time devoted to address- 
ing them, and the number of students removed from the classroom. Given the year-to-year vari- 
ability in suspension counts, a fuller analysis of Project GRAD’s effects on the number of sus- 
pensions (a measure for which historical data are available) will be deferred until the second year 
of post-implementation data are available. The material presented in this section indicates the 
baseline trend against which future levels will be compared. 

• Over the course of the first year of Project GRAD Newark implementa- 
tion, there has been a substantial decline in the number of discipline re- 
ferrals to the principal’s office among students in the elementary and 
middle grades at Project GRAD Newark schools. 

Information from two different data sources support this conclusion. Teachers responding 
to the MDRC survey in January 1999 reported fewer discipline referrals in the 1998-99 school 
year than in the previous year. Averaged across all survey respondents, the number of referrals 
per month dropped from 3.1 in the 1997-98 school year to 1.8 in the 1998-99 school year, a re- 
duction of about 40 percent. Declines were observed in every school and at most grade levels 
surveyed (grades K-8 were surveyed). 

CMCD data show a similar decline in discipline referrals over the course of the 1998-99 
school year.^ Across the seven elementary and middle schools where CMCD was implemented 
over the course of this entire year (Dayton Street School is not included because it joined Project 
GRAD mid-year), discipline referrals totaled 450 in the first half of the school year (September 

^These referral counts are presented in Freiberg, 1999. 
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1998-January 1999), with a monthly average of 90. During the second half of the school year 
(February 1999- June 1999), in contrast, referrals totaled 236, with a monthly average of 47. At 
the one school for which detailed historical records on discipline referrals prior to the launch of 
Project GRAD were available (Belmont Runyon), the decline was especially precipitous — from 
149 in the 1997-98 school year to 25 in the 1998-99 school year.^ 

• Superintendent suspensions of students in grades Pre-K-8 in Project 
GRAD Newark schools rose and then fell over the five-year period shown 
in Figure 4.1, while suspensions by principals showed a gradual decline 
over the same period. 

Superintendent suspensions are typically responses to serious discipline problems. As 
shown in Figure 4.1, the number of superintendent suspensions in Project GRAD Newark 
schools rose from two in the 1994-95 school year to 21 in the 1996-97 school year and then de- 
clined to 15 in the 1997-98 school year and seven in the 1998-99 school year. A similar pattern 
of superintendent suspensions was observed in the elementary and middle comparison schools, 
while in the rest of the district the number of superintendent suspensions remained stable in the 
1998-99 school year. This suggests that the recent changes in the number of superintendent sus- 
pensions in the Project GRAD Newark schools may be part of similar changes throughout SLT 
TTT, from which both Project GRAD and comparison elementary and middle schools are drawn. 
Principal suspensions, which are meted out for less serious discipline problems, totaled 41 1 in 
the 1994-95 school year and 283 in the 1997-98 school year and then declined to 243 in the 
1998-99 school year. There was also a decline in principal suspensions in the rest of the district 
from the 1996-97 school year to the 1998-99 school year, suggesting that the factors producing 
the decline in this measure are not unique to the Project GRAD Newark schools. 

• The number of superintendent suspensions of students increased at Mal- 
colm X Shabazz High School in the 1998-99 school year, while principal 
suspensions showed a less pronounced upward trend. 

Over the first four years shown in Figure 4.1, the number of superintendent suspensions 
at Shabazz High School remained relatively stable until it increased in 1998-99. A similar, 
though less pronounced, increase in the number of superintendent suspensions was also observed 
in the comparison high school, but the total for the rest of the district declined in the 1998-99 
school year. The number of principal suspensions showed a slight upward trend at Shabazz over 
the last three years shown in the figure. The number of principal suspensions increased in the 
comparison high school and in schools in the rest of the district over the same period. 

Measures of Student Achievement 

One of the central objectives of Project GRAD is to improve student achievement in the 
elementary and middle grades (thereby preparing students to take advantage of the GRAD schol- 



*As mentioned earlier, one component of CMCD is a better system for counting and analyzing discipline refer- 
rals. Prior to the start of Project GRAD Newark, information on the number and type of referrals was not consis- 
tently collected. 
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Figure 4.1 

Trends in Student Suspensions for Project GRAD Newark Schools 
and Schools in the Rest of the District, 1994-99 



SUPERINTENDENT SUSPENSIONS 




111994-95 

^1995-96 

■ 1996-97 
□ 1997-98 

■ 1998-99 




School(s) 




SOURCE: These data were obtained from the Newark Public Schools Office of Student Information Services. 
NOTES: Missing bars represent zero suspensions. 

The comparison elementary and middle schools are other SLT III schools not in Project GRAD Newark, 
and the comparison high school is Weequahic. 





arship offer) and in high school (leading to higher rates of high school graduation and college 
attendance). Determining whether student achievement has improved may seem straightforward, 
but the evolving nature of state and district testing practices complicates the achievement analy- 
sis. Moreover, disentangling the contribution of Project GRAD from that of other school reforms 
adds another layer of complexity. At this stage in the MDRC evaluation, the task is to track re- 
cent trends, but this discussion also describes a simplified version of the method for estimating 
the contribution of Project GRAD Newark to observed improvements in achievement that will be 
used in future reports and presents an illustration of this method. 

Overview of Testing Undertaken in Newark Public Schools 

Table 4.2 lists, by grade and school year, the tests annually administered to students in the 
Newark public schools.’ As shown, the 1990s have seen rapid change in the type of tests used. 
The coming years promise further evolution as tests are made to align more closely with state 
core curriculum content standards. Thus, the measures of student achievement that are available 
will change over time at many grade levels, preventing straightforward analysis of scores in a 
particular grade over time. The system for monitoring change in student achievement for Project 
GRAD Newark must take account of the reality of this dynamic context. The MDRC evaluation 
will be able to track student test scores over time, but the analysis of the role of Project GRAD in 
causing any changes in test scores will focus specifically on grades for which there is sufficient 
stability in the tests administered to permit a detailed analysis of trends. 

Summary of Scores on State Assessments 

Table 4.3 shows several summary measures of individual Project GRAD Newark 
schools’ test scores on the new state assessments introduced during the 1998-99 school year for 
4***- and 8**'-grade students.® Figures 4.2 and 4.3 present summary measures for the ll‘* *’-grade 
High School Proficiency Test (HSPT) in the Project GRAD Newark schools and schools in the 
rest of the district. (Some of these data were presented earlier in this report.) All these data 
should be seen as defining a baseline against which future student achievement can be compared. 
The introduction of new 4***- and 8‘*’-grade tests in New Jersey in 1998-99 precludes a time-trend 
analysis at this point, and, as mentioned earlier, 1 1*** graders who took the HSPT in 1998-99 were 
not eligible for Project GRAD Newark. The most notable lesson to draw from Table 4.3 and Fig- 
ures 4.2 and 4.3 is that only a low percentage of students in the Project GRAD Newark schools 
had demonstrated attainment of state standards on any of these tests by the 1998-99 school year; 
the need for future improvement is clear. These data vividly illustrate why the Shabazz feeder 
pattern was chosen as the site of Project GRAD Newark: These schools face the types of educa- 
tional challenges that Project GRAD is designed to address. 



^The district also administers a separate test (Aprenda) to Spanish-speaking students. In the 1996-97 school 
year, the ninth edition of the Stanford Achievement Test (commonly known as the SAT9) replaced the eighth edition 
(SAT8), but the test developer provided a series of tables “equating” scores on the eighth and ninth editions of the 
test to allow for comparisons between years. 

*In this school year, a new 4‘*'-grade test (the Elementary School Proficiency Assessment; ESPA) was intro- 

duced, and the Grade Eight Proficiency Assessment (GEPA) replaced the Early Warning Test (EWT). 
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Student Achievement Tests Administered in Grades 2-11 in Newark Public Schools 
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Table 4.3 

Scores on the 4“*- and 8“'-Grade State Assessments for Project GRAD Newark Schools 
and Schools in the Rest of the District and State, 1998-99 
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Figure 4.2 

Reading and Writing Scores on the High School Proficiency Test (HSPT) for Shahazz 
High School and Schools in the Rest of the District and State, hy School Year 




School(s) 



□ 1997-98 
■ 1998-99 




School(s) □ 1997-98 

■ 1998-99 




SOURCES: These data were obtained from Philadelphia Online's School Report Cards (1998), "New Jersey School Data," 
<http://www.philly.com/paclcages/njschools/njs98.asp/>; New Jersey Department of Education (1999b), October 1998 
High School Proficiency Test (HSPT): State Summary. Trenton, NJ; and the New Jersey Department of Education Web 
site at http://www.state.nj.us/cgi-bin/education/hsptsearch99.pl. 

NOTES: These data are based on the fall administration of the High School Proficiency Test (HSPT). First-time test- 
takers and retested students are included, and special education students and students with limited English proficiency are 
excluded. 

The District Factor Group (DFG) is a measure of income, education attainment, and other demographic characteristics 
of district residents. It ranges from A in the poorest districts to I and J in the wealthiest. Newark is in DFG-A. 



Ml 

-68- II 




Figure 4.3 



Math and Total Scores on the High School Proficiency Test (HSPT) for Shahazz 
High School and Schools in the Rest of the District and State, hy School Year 



MATH 




School(s) 



01996-97 
□ 1997-98 
01998-99 



ALL SECTIONS 




School(s) 



01996-97 
□ 1997-98 
01998-99 



SOURCES: These data were obtained from Philadelphia Online's School Report Cards (1998), "New Jersey School Data," 
<http://www.philly.com/packages/njschools/njs98.asp/>; New Jersey Department of Education (1999b), October 1998 
High School Proficiency Test (HSPT): State Summary. Trenton, NJ; and the New Jersey Department of Education Web 
site at http://www.state.nj.us/cgi-bin/education/hsptsearch99.pl. 



NOTES: These data are based on the fall administration of the High School Proficiency Test (HSPT). First-time test- 
takers and retested students are included, and special education students and students with limited English proficiency are 
excluded. 

The District Factor Group (DFG) is a measure of income, education attainment, and other demographic characteristics 
of district residents. It ranges from A in the poorest districts to I and J in thdj^01thiest. Newark is in DFG-A. 
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Analysis of Trends in Stanford Achievement Test Scores 



It is possible to conduct a more detailed time-trend analysis of student scores on the Stan- 
ford Achievement Test, which is administered in many of the other grades. This section presents 
data on changes in test scores over time and an early assessment of whether there are signs that 
Project GRAD is causing an improvement in student achievement. The current findings for test 
scores in 2"^- and 3"*-grade reading and math and in 9“’- and 10“’-grade math are presented, and a 
simplified version of the approach that will be taken in future analyses to disentangling the con- 
tribution of Project GRAD to improvements in scores is illustrated.^ 

The first step in the analysis is the same whether the goal is to track changes in test scores 
to see if improvement is occurring in the Project GRAD Newark schools or to determine whether 
Project GRAD Newark caused the observed improvement: Test scores are compiled for the peri- 
ods before and after Project GRAD Newark implementation. If the goal is assessing whether Pro- 
ject GRAD Newark has caused the observed improvement, the scores in the years after program 
implementation began need to be compared to what would have happened in the absence of Pro- 
ject GRAD Newark. Two methods of estimating what would have happened in the absence of the 
program are used; comparisons are then made between each of these baselines and what actually 
happened in the Project GRAD Newark schools. These comparisons yield estimates of the differ- 
ence that Project GRAD Newark made. 

Comparison #1: Use the Project GRAD Newark schools’ test score histories to estimate 
what to expect in Project GRAD Newark schools once the program starts. 

If, once Project GRAD Newark is operating, the test scores observed in the Project 
GRAD Newark schools exceed what would be expected on the basis of those schools’ score his- 
tory, then Project GRAD may be helping to raise student test scores at the schools.'” If this crite- 
rion is met, the analysis moves to a second comparison: 

Comparison #2; Use the trends in test scores of comparison schools where Project 
GRAD Newark is not being implemented to find out if any observed improvement is 
likely to be caused by Project GRAD. 

The more similar the comparison schools are to the Project GRAD Newark schools (ex- 
cept for the presence of the program), the better the trends in their test scores will represent what 



’For the grades analyzed, test scores for the same or similar subject matter scales (or subscales) are reported 
over the period prior to Project GRAD Newark implementation. For all other grades and subjects, testing practices 
have changed enough during the years before Project GRAD Newark and/or during the period of its implementation 
to rule out the use of this type of analysis to assess Project GRAD Newark’s role in causing changes in test scores. 

'°In statistical terms, the pre-reform test scores are used to estimate a pre-reform trend in scores; this trend is 
then extrapolated into the post-reform period as the best estimate of what would have happened in the absence of 
Project GRAD. If the actual test scores in the post-reform period are higher than this predicted level by a statistically 
significant amount, then it is reasonable to conclude that there has been an improvement in test scores. The same 
approach can be used to find post-reform scores that lie below the expected level. See Bloom (1999) for a discussion 
of the statistical techniques used. 
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would have happened in the Project GRAD Newark schools in the absence of the program.*' If 
the comparison schools show no comparable boost in scores above the expected level, then the 
positive impact is likely to be a product of Project GRAD. 

The reality of estimating program impacts in Newark will clearly be complex because (1) 
no schools perfectly “match” the Project GRAD Newark schools; that is, no schools are identical 
to the Project GRAD Newark schools in every respect except the presence of the program; and 
(2) all the comparison schools will be implementing some version of whole-school reform over 
the coming years. Thus, the analysis can provide supporting evidence for, but not definitive proof 
of. Project GRAD’S positive impacts.'^ 

• Although Project GRAD Newark implementation is at an early stage, and 
neither the reading nor the math curricular component had been imple- 
mented before the test data presented here were collected, there is already 
strong evidence that 3'‘*-grade total math scores in program schools are 
higher than would be expected from their own past score histories, and 
there is suggestive evidence of a similar break from trend in 3'‘*-grade to- 
tal reading scores. 

Figure 4.4, which is explained in detail in Box 4.1, illustrates how this analysis was per- 
formed for 3‘^‘*-grade total math scores on the Stanford Achievement Test. The scores of Project 
GRAD Newark schools in school years before program implementation were used to estimate an 
expected trend in, or predicted level of, test scores during the implementation period. The aver- 
age total math test score for 3”* graders in the 1998-99 school year is depicted in Figure 4.4 as a 
small circle. It falls above the expected trend in scores by a statistically significant amount.'^ 
This positive deviation from the expected math score provides evidence that average 3'^‘*-grade 
total math scores in the Project GRAD Newark schools in the 1998-99 school year were higher 
than expected based on these schools’ past history. A similar analysis of 3'^‘*-grade total reading 
scores found that the scores in 1998-99 fell above the expected trend line, but not by a statisti- 
cally significant amount. Nevertheless, this finding is suggestive of a break in the trend for read- 
ing test scores. Again, it is important to note that this analysis covers a period prior to the imple- 
mentation of the curricular components of Project GRAD. Possible explanations for these results 
are presented at the end of this chapter. 

• There is evidence that this break from trend is driven by reforms in the 
Project GRAD Newark schools rather than by other reforms in SLT III 
or the district. The comparison schools did not have better-than-expected 
3’^'' -grade math test scores in the 1998-99 school year, and the small in- 
crease in the rest of the district is not statistically signiHcant. 



"For elementary and middle school students, the analysis compares the Project GRAD schools as a group to the 
comparison schools as a group. For high school students, trends in outcomes for students at Shabazz High School are 
compared to those for students at Weequahic High School and for students at other high schools in the district. 

'^This type of analysis can also be done for other student outcomes. Subsequent reports will report findings 
based on this method. 

"in other words, it lies outside the confidence interval around the expected trend calculated using its mar- 
gin of error. 
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Box 4.1 

Time-Trend Analysis 



Figure 4.4 illustrates the time-trend approach used to assess Project GRAD Newark’s 
impacts on scores on the ninth edition of the Stanford Achievement Test (SAT9). It uses the 
case of 3”*-grade total math scores on the ninth edition of the scores, where there is strong evi- 
dence of an improvement in test scores in Project GRAD Newark schools during the first year 
of program implementation (1998-99). The analysis presents average total math tests scores for 
all 3”* graders tested in the Project GRAD Newark schools in each of the years shown on the 
horizontal axis. Testing was done in the spring of each year: 1993 refers to tests administered 
at the end of school year 1992-93, 1994 to tests administered at the end of school year 1993- 
94, and so forth. The vertical axis shows the average test scores, expressed in normal curve 
equivalents — a standardized measure that facilitates statistical analyses of changes over time. 

As the figure shows, total math scores in the Project GRAD Newark schools have been 
fairly stable — although have declined slightly — over the six school years prior to the intro- 
duction of Project GRAD Newark. These scores are indicated by triangles. In general, the av- 
erage scores fall below national norms. In school year 1997-98, for example, the average total 
math score for 3”* graders in Project GRAD schools, expressed in normal curve equivalents, 
was 40.8. This translates into a percentile score of approximately 33, indicating that the New- 
ark average test score was better than only 33 percent of all test takers nationally. 

The trend line in Figure 4.4 was estimated from the actual average test scores from 1993 
to 1998 (using time-series linear regression techniques) and then extrapolated into the period of 
Project GRAD Newark implementation (from 1999).’ This trend serves as a benchmark against 
which test scores can be compared once Project GRAD is in place. It represents the expected 
test scores based on the past history of scores in Project GRAD Newark schools. In the Project 
GRAD years (1999 and after), a confidence interval is shown around the expected trend line in 
the figure, much as survey results are reported with a margin of error.^ From 1999 onward, av- 
erage test scores that fall within this confidence interval are statistically indistinguishable from 
the estimated trend, while those falling outside the confidence interval show a statistically sig- 
nificant deviation from the trend. The 1999 average total math test score for 3”* graders is 
shown as a small circle. It falls above the estimated trend line and outside the confidence inter- 
val around the estimated trend. Therefore, this positive deviation from the trend is statistically 
significant and provides strong evidence of a break from the past pattern of scores for this topic 
and grade. 

^One assumption of this approach is that a straight-line Oinear) trend is a reasonable estimate of 
the existing pattern of test scores. For the period 1993-98, this appears to be a reasonable assumption 
about the Stanford Achievement Test scores. 

^e confidence interval was calculated as a two-tailed t-test at the .05 level. 
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Figure 4.4 



Average 3*^^-Grade Math Scores for Project GRAD Newark Schools 
on the Stanford Achievement Test, by School Year 
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Expected 
Trend Line 



Confidence 
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SOURCE: These data were obtained from the Newark Public School District Office. 
NOTE: The circle represents the actual 1998-99 score. 
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The average 3'^‘*-grade total math scores in the comparison schools and in the whole dis- 
trict’s schools were examined to see if in the 1998-99 school year they exhibited the same break 
from the past trend. If they did, then the 3”*-grade math test score improvement observed in the 
Project GRAD Newark schools in the 1998-99 school year is likely attributable to other initia- 
tives and reforms operating in SLT III or the district. Figure 4.5 shows that a break from trend 
such as that in the Project GRAD Newark schools was not observed in the comparison schools 
and that the small increase above trend in the rest of the district is not statistically significant. 
This is consistent with the conclusion that the boost in average math scores in the Project GRAD 
Newark schools is driven by something unique to this set of schools.’'* 

• The distribution of 3'^‘*-grade math test scores in 1998-99 suggests that the 
boost in scores was spread fairly evenly across the students in the Project 
GRAD Newark schools. The break from trend was produced by a decline 
in the number of very low scores and an increase in the number of scores 
above the 50“* percentile. 

Time trends were estimated for the percentage of 3"^ graders whose total math test scores 
fell in the bottom 25 percent of the distribution of all test takers nationally (that is, at the 25*** 
percentile or below) and for the percentage whose scores fell at or above the 50*** percentile. In 
the 1998-99 school year, there was a statistically significant break from the past trends on both 
measures of performance on the math test: Compared with the previous five years, significantly 
fewer 3“* graders in Project GRAD Newark schools scored very low, and significantly more 
scored above the 50*** percentile. In reading, in contrast, the test score improvement resulted from 
an increase in the number of students scoring above the 50*** percentile, an effect driven primarily 
by a very large and statistically significant deviation from trend in one school. 

• In grades 2, 9, and 10, there were no positive breaks in the 1998-99 school 
year from the past trends in test scores. 

There were no positive deviations from the expected trend in average test scores in 2"'*- 
grade reading and math or in 9***- or 10‘*'-grade math. There was, however, one negative finding in 
these grades, namely, in 9‘*'-grade math at Shabazz High School. There, scores were lower than 
expected based on past history (that is, lay below the trend line) by a statistically significant 
amount. The comparison high school (Weequahic High School) and all other high schools in the 
district also had lower-than-expected scores based on past history, but the deviations from trend 
at those schools was less pronounced than at Shabazz. This finding highlights the already identi- 
fied need to strengthen instruction at the Project GRAD Newark high school to support students’ 
efforts to take advantage of the GRAD scholarship offer, for which students who were in grade 9 
in the 1998-99 school year are eligible. 

• The program’s impacts on 3'‘‘’-grade test scores could be related to the 
combination of the district’s focus on the 3”^** grade as crucial to success 



'■‘a formal statistical test was also performed that showed that the difference between the deviation from the es- 
timated trend in the Project GRAD Newark schools and the deviations from the estimated trends in the comparison 
schools and the rest of the district was statistically significant. 
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Figure 4.5 

Trends in 3*^^-Grade Average Math Scores for Project GRAD Newark Schools, 
Comparison Schools, and Schools in the Rest of the District 
on the Stanford Achievement Test, hy School Year 
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SOURCE: These data were obtained from the Newark Public School District Office. 
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on the new 4***-grade state assessment and to some components of Project 
GRAD Newark (CMCD and CIS). 

The test scores analyzed here were observed in the first year of Project GRAD Newark 
implementation. Because it often takes several years for a reform’s effects on test scores to 
emerge, the lack of pervasive impacts is not surprising. 

At this early stage in the MDRC evaluation, the observed improvements in S’^^'-grade test 
scores might stem from various sources. First, as discussed earlier, students in the early grades 
received a more concentrated “dose” of program services during this stage of Project GRAD 
Newark implementation than did those in the high school. Further, throughout SLT in, teachers 
were focused on the new 4***-grade state test (the Elementary School Proficiency Assessment) and 
viewed improvement in student achievement in the 3"* grade as crucial to boosting future per- 
formance on the 4*-grade test.’^ Finally, in the Project GRAD Newark schools, CMCD and CIS 
might have improved the learning environment and thereby raised test scores in the Project 
GRAD Newark schools in the 1998-99 school year relative to scores in the rest of SLT in and the 
district as a whole. 

Challenges Ahead 

At this early stage of Project GRAD Newark implementation, analysis of the real effects 
of the program must await fuller implementation of its components across the feeder pattern of 
schools and the passage of more time in which the program can change student outcomes. There 
are some encouraging test score effects at one grade level, which are likely driven by a combina- 
tion of the district’s instructional emphasis on this grade and improvements in the learning envi- 
ronment stemming from some Project GRAD Newark components. Clear messages from the pre- 
sent analysis of trends include the relatively low current level of the Project GRAD Newark 
schools with respect to many student outcomes — underscoring the reasons for choosing these 
schools for the initiative — and the importance of Project GRAD to the teachers and students as 
a means of making the needed improvements. This analysis also highlights the need for strong 
program implementation and continued strengthening of the curriculum, as well as the signifi- 
cance of schools’ and funders’ long-term commitment to Project GRAD Newark. 



'^Because 1998-99 was the first school year in which the state’s 4*'’-grade test was administered, it is not possi- 
ble to investigate whether there was a break from the past trend in test scores at this grade level. 
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Table A.l 

Key Characteristics of Comparison Schools in the Project GRAD Newark Evaluation, 1998-99 



a 

o 

o 



a> X 

& c 



§ 
S’ § 



C u 

o 3 
•S g 

cd < 

n: 



^ > 
n S 



G S> 

o 2 
ts c 

G O 

;g > 

U < 



o 

X 

4> 

U G 



8 G 

§ I 

,G < 

u 



2 ^ 

OQ CO 



^ 3 
3 3 

^ - 



_ > 
OQ < 



ir> 

^ On 

ON 00 









O 

CN 



00 On 
JL 

cn Tj- 



00 NO 



o 






CO O 

^ $ 






CO 



^ On 



ON 

CO 



G 



O 



o 



o 



o 



o 



o 



o 



o 



o 






G 

Q> 

*o 

tG 

O 



ON 

uo 



CO 

00 



s 

00 



tJ- 

00 






Tf 

On 



CO 

00 



00 



CO 

00 



So 

r*^ 



o 

G 

3 



o .3 



On 



NO 



c 

o 



o 


•o 

o> 


•S 

'o 


CO 

'W) 




.o 

‘3a 


l-l 

1 

•o 


*G 

G 

O 




U5 

‘C 

o> 


t 

0> 

CO 


0> 


c 


CO 

C 

0> 

•o 




o 

u 

3 


3 

•o 

o 


CO 

4»» 

G 

0> 

•o 


4»» 


CO 

o 

•o 

G 


c 

o 

•o 


•o 

<D 

4»» 


G 

O 

•o 


•o 

oa 


‘o 

0) 




u 


3 


.3 


3 


3 


p 




3 


u 


o 


55 


hJ 


to 


55 


vh 


CO 


to 



cs 

oo 



'o s; 



CO 



VO 



O 

o 



o 

<N 



^ s ^ 

CO 5; <N 



<N 

On 



^ ^ .o 

^ SD 

On 



SC> <N 

On 



o S 



On 



CO 

On 



NO 



CO 



ON 

CO 



^ - 
O CO 
rt ^ 



NO 

CO 



ON 

cs 



NO 

Tl- 



NO 



^ t: 

VO CO 
CO ^ 



CO 

CO 



.2 

Cd 



CO 



o 

CO 



cs 



CO 



CO 



CM 

CM 



CO 

CO 



On 



O 

NO 





CM 


'o- 


’t; 


*S^ 

VJ" 


CO 


CM 




CM 


VO 


On 


CO 




CM 



to 


CO 




to 




to 


G 

t-N 


CO 

CO 

G 


"^to 


JS 

o 


5 


to 

& 


to 

u 

g 

•o 

f5 


*0 

to 

to 

G 


Cd 

u 


G 

to 

1 

c 

to 


s 

CO 

t-N 

to 

43 


to 

T3 

CO 

'l-N 

to 


to 

< 


to 

> 

< 


'£ 

o 


-o 

3 

w 

CO 


O 

G 

to 

H 


4»» 

CO 

G 

E 



G 

O 

•o 

3 

•W 

CO 



o 

o 

s 

O 



o 

o 

*G 

o 

CO 

o 

z 

3 

cu 

t 

•o 

G 

3 

O^ 

On 

On 



1 

U 

G 

o 

D< 

o 

VC 

"o 

o 

o 

CO 

CO 

'oj 

.s 

"g 

o 

3 

Sr 

•o 

3 



cu 

e 

o 

1? 



B 

x> 

o 



o 

3 

w 

3 

•o 



C o 

^ CO 

CO § 
W *2 
U ^ 
£ 

^ o 
O ‘s 

CO 



o 

>* 

Cd 

:S 

to 

G 



•o 

S 

Im 

O 

•g 



G 

O 



o 

>* 

o 

s 

G 

§ 



G p 
2 § 
2 IS 



O 

•ts 

o 

•o 



o 

o 



0 

S 

G 

•o 

s 

"o 

« ij 

1 « 

g ;3 

2 ^ 

2 e- 

to G 



^ = o 

S I S 

5 ° t 2 

PQ bo 



- CO 

fS S g 



CO ^ 

s- 

O 

:z; 



§ ^ 

O CO 

o ^ 



o 

ERIC 



- 78 - 



References 



Association for Children of New Jersey. 1999. Newark Kids Count 1998: A Profile of Child Well- 
Being. Newark, NJ: Association for Children of New Jersey. 

Bloom, Howard S. 1999. Estimating Program Impacts on Student Achievement Using "Short" 
Interrupted Time Series. New York: Manpower Demonstration Research Corporation. 

Bodilly, Susan J. 1998. Lessons from New American Schools’ Scale-Up Phase: Prospects for Bringing 
Designs to Multiple Schools. Santa Monica, CA: RAND. 

Education Trust. 1999. “Ticket to Nowhere: The Gap Between Leaving High School and Entering 
College and High Performance Jobs.” Thinking K-16 3, 2 (Fall): 1-31 . 

Freiberg, Jerome. 1996. “Creating a Climate for Learning.” Educational Leadership 54, 1 (September): 
32-36. 



. 1999. Consistency Management and Cooperative Discipline Newark Report, 1998-1999. 

Houston, TX: University of Houston. 

Furman, Susan H. 1994. “Challenges in Systemic Education Reform.” CPRE Policy Briefs (September 
14): 1-10. 

Glennan, Thomas, Jr. 1998. New American Schools After Six Years. Santa Monica, CA: RAND. 

Harcourt Brace & Company. 1997. Stanford Achievement Test Series, Ninth Edition: Spring Norms 
Book. San Antonio, TX: Harcourt Brace & Company. 

Kemple, James, and Jason Snipes. 2000. “Talent Development Comprehensive School Reform Model 
Evaluation Design.” New York: Manpower Demonstration Research Corporation. 

King, Bruce M., and Fred M. Newmann. 2000. “Will Teacher Learning Advance School Goals?” Phi 
Delta Kappan 81,8 (April): 576-580. 

Malcolm X Shabazz High School. 1998. Malcolm X Shabazz High School: School Profile, 1998-99. 
Newark, NJ: Malcolm X Shabazz High School. 

National Commission on Excellence in Education. 1983. A Nation at Risk. Washington, DC: 
Government Printing Office. 

Newark Public Schools. 1999. Raising Quality and Standards: The 1998-2003 Strategic Plan. Newark, 
NJ: Newark Public Schools. 

New Jersey Department of Education. 1999. May 1999 Elementary School Proficiency Assessment, 
STATE SUMMARY. Trenton, NJ. 

. 1999a. March 1999 Grade Eight Proficiency Assessment (GEPA), STATE SUMMARY. Trenton, 

NJ. 

. 1999b. Octoberl998 High School Proficiency Test (HSPTll), STATE SUMMARY. Trenton, NJ. 

O’Day, Jennifer, Margaret Goertz, and Robert Floden. 1995. “Building Capacity for Education 
Reform.” CPRE Policy Briefs (December 18): 1-13. 



ERIC 



- 79 - 



Opuni, Kwame. 1998. Project GRAD: Graduation Really Achieves Dreams, 1997-98, Evaluation 
Report. Houston, TX: University of Houston. 

Philadelphia Online’s School Report Cards. 1999. “New Jersey School Data”: 
http://www.education.philIy.com/tests/njreport99/. 

Philadelphia Online’s School Report Cards. 1998. “New Jersey School Data”: 
http://www.philIy.com/packages/njschools/njs98.asp/. 

Rallis, Sharon, and Nancy Zajano. 1997. “Keeping the Faith Until the Outcomes Are Obvious.” Phi 
Delta Kappan 78, 9: 706-709. 

School Leadership Team m. 1999. Data Foundation Report: Avon Avenue School, 1999-2000. Newark, 
NJ: Newark Public Schools. 

. 1999a. Data Foundation Report: Belmont Runyon School, 1999-2000. Newark, NJ: Newark 

Public Schools. 

. 1999b. Data Foundation Report: Dayton Street School, 1999-2000. Newark, NJ: Newark Public 

Schools. 

. 1999c. Data Foundation Report: Louise A. Spencer School, 1999-2000. Newark, NJ: Newark 

Public Schools. 

. 1999d. Data Foundation Report: Madison Avenue School, 1999-2000. Newark, NJ: Newark 

. Public Schools. 

. 1999e. Data Foundation Report: Peshine Avenue School, 1999-2000. Newark, NJ: Newark 

Public Schools. 

. 1999f. Data Foundation Report: William H. Brown Academy, 1999-2000. Newark, NJ: Newark 

Public Schools. 

Slavin, Robert E. 1997 . Sand, Bricks, and Seeds: School Change Strategies and Readiness for Reform. 
Baltimore, MD: Center for Research on the Education of Students Placed at Risk, Johns Hopkins 
University. 

U.S. Department of Education. 1998. Guidance on the Comprehensive School Reform Demonstration 
Program. Washington, DC: U.S. Department of Education. 




SO 

-80- 



Selected Publications on MDRC Projects 



Note: For works not published by MDRC, the publisher’s name is shown in parentheses. A complete publications list 
is available from MDRC and on its Web site (www.mdrc.org), which also contains copies of MDRC’ s 
publications. 

Education Reform 

Project GRAD 

This evaluation examines Project GRAD, an education initiative targeted at urban schools and combining a number 
of proven or promising reforms. 

Career Academies 

The largest and most comprehensive evaluation of a school-to-work initiative, this study examines a promising 
approach to high school restructuring and the school-to-work transition. 

Career Academies: Early Implementation Lessons from a 1 0-Site Evaluation. 1996. James Kemple, Jo Ann Leah 
Rock. 

Career Academies: Communities of Support for Students and Teachers — Emerging Findings from a 1 0-Site 
Evaluation. 1997. James Kemple. 

Career Academies: Building Career Awareness and Work-Based Learning Activities Through Employer 
Partnerships. 1999. James Kemple, Susan Poglinco, Jason Snipes. 

SchooI-to-Work Project 

A study of innovative programs that help students make the transition from school to work or careers. 

Home-Grown Lessons: Innovative Programs Linking School and Work (Jossey-Bass Publishers). 1995. Edward 
Pauly, Hilary Kopp, Joshua Haimson. 

Home-Grown Progress: The Evolution of Innovative School-to-Work Programs. 1997. Rachel Pedraza, Edward 
Pauly, Hilary Kopp. 

Project Transition 

A demonstration program that tested a combination of school-based strategies to facilitate students’ transition from 
middle school to high school. 

Project Transition: Testing an Intervention to Help High School Freshmen Succeed. 1999. Janet Quint, Cynthia 
Miller, Jennifer Pastor, Rachel Cytron. 

Equity 2000 

Equity 2000 is a nationwide initiative sponsored by the College Board to improve low-income students’ access to 
college. The MDRC paper examines the implementation of Equity 2000 in Milwaukee Public Schools. 

Getting to the Right Algebra: The Equity 2000 Initiative in Milwaukee Public Schools. 1999. Sandra Ham, Erica 
Walker. 

Career Beginnings 

An evaluation of a program that seeks to increase college attendance and improve job quality among disadvantaged 
high school students. 

Career Beginnings Impact Evaluation: Findings from a Program for Disadvantaged High School Students. 1990. 
George Cave, Janet Quint. 



Effects of Welfare and Antipoverty Programs on Children 

Minnesota Family Investment Program 

An evaluation of Minnesota’s pilot welfare reform initiative, which aims to encourage work, alleviate poverty, and 
reduce welfare dependence. 

Reforming Welfare and Rewarding Work: Final Report on the Minnesota Family Investment Program. Volume 2: 
Effects on Children. 2000. Lisa Gennetian, Cynthia Miller. 

Reforming Welfare and Rewarding Work: A Summary of the Final Report on the Minnesota Family Investment 
Program. 2000. Virginia Knox, Cynthia Miller, Lisa Gennetian. 
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Canada’s Self-Sufficiency Project 

A test of the effectiveness of a temporary earnings supplement on the employment and welfare receipt of public 
assistance recipients. 

The Self-Sufficiency Project at 36 Months: Effects on Children of a Program That Increased Parental Employment 
and Income (Social Research and Demonstration Corporation, Ottawa, Canada). 2000. Pamela Morris, Charles 
Michalopoulos. 

National Evaluation of Welfare-to-Work Strategies 

Conceived and sponsored by the U.S. Department of Health and Human Services, with support from the U.S. 
Department of Education, this is the largest-scale evaluation ever conducted of different strategies for moving people 
from welfare to employment. 

Do Mandatory Welfare-to-Work Programs Affect the Well-Being of Children? A Synthesis of Child Research 
Conducted as Part of the National Evaluation of Welfare-to-Work Strategies (HHS/ED). 2000. Gayle Hamilton. 



Teen Parents on Welfare 

Teenage Parent Programs: A Synthesis of the Long-Term Effects of the New Chance Demonstration, Ohio’s 
Learning, Earning, and Parenting (LEAP) Program, and the Teenage Parent Demonstration (TPD). 1998. Robert 
Granger, Rachel Cytron. 

Ohio’s LEAP Program 

An evaluation of Ohio’s Learning, Earning, and Parenting (LEAP) Program, which uses financial incentives to 
encourage teenage parents on welfare to stay in or return to school. 

LEAP: Final Report on Ohio’s Welfare Initiative to Improve School Attendance Among Teenage Parents. 1997. 
Johannes Bos, Veronica Fellerath. 

New Chance Demonstration 

A test of a comprehensive program of services that seeks to improve the economic status and general well-being of a 
group of highly disadvantaged young women and their children. 

New Chance: Final Report on a Comprehensive Program for Young Mothers in Poverty and Their Children. 1997. 
Janet Quint, Johannes Bos, Denise PoliL 

Parenting Behavior in a Sample of Young Mothers in Poverty: Results of the New Chance Observational Study. 
1998. Martha Zaslow, Carolyn Eldred, editors. 



Other Programs for Youth 

JOBSTART Demonstration 

A test of a program combining education, training, support services, and job placement for very disadvantaged 
young high school dropouts. 

JOBSTART: Final Report on a Program for School Dropouts, 1993. George Cave, Hans Bos, Fred Doolittle, Cyril 
Toussaint. 
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A test of a school-conditioned job guarantee for low-income youth. 

Lessons from a Job Guarantee: The Youth Incentive Entitlement Pilot Projects. Monograph. 1984. Judith Gueron. 



MDRC Working Papers on Research Methodology 

A new series of papers that explore alternative methods of examining the implementation and impacts of programs 
and policies. 

Estimating Program Impacts on Student Achievement Using "Short'* Interrupted Time Series. 1999. Howard 
Bloom. 

Using Cluster Random Assignment to Measure Program Impacts: Statistical Implications for the Evaluation of 
Education Programs. 1999. Howard Bloom, Johannes Bos, Suk-Won Lee. 
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About MDRC 



The Manpower Demonstration Research Corporation (MDRC) is a nonprofit, 
nonpartisan social policy research organization. We are dedicated to learning what 
works to improve the well-being of low-income people. Through our research and 
the active communication of our findings, we seek to enhance the effectiveness of 
social policies and programs. MDRC was founded in 1974 and is located in New 
York City and San Francisco. 

MDRC’s current projects focus on welfare and economic security, education, and 
employment and community initiatives. Complementing our evaluations of a wide 
range of welfare reforms are new studies of supports for the working poor and 
emerging analyses of how programs affect children’s development and their 
faimlies’ well-being. In the field of education, we are testing reforms aimed at 
improving the performance of public schools, especially in urban areas. Finally, our 
community projects are using innovative approaches to increase employment in 
low-income neighborhoods. 

Our projects are a mix of demonstrations - field tests of promising program models 

- and evaluations of government and community initiatives, and we employ a wide 
range of methods such as large-scale studies to determine a program’s effects, 
surveys, case studies, and ethnographies of individuals and families. We share the 
findings and lessons from our work - including best practices for program operators 

- with a broad audience within the policy and practitioner community, as well as the 
general public and the media. 

Over the past quarter century, MDRC has worked in almost every state, all of the 
nation’s largest cities, and Canada. We conduct our projects in partnership with state 
and local governments, the federal government, public school systems, community 
organizations, and numerous private philanthropies. 
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New York, NY 10016 
(212) 532-3200 

www.mdrc.org 

88 Kearny Street, Suite 1800 
San Francisco, GA 94108 
(415) 781-3800 
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